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Preface 


The  Health  Care  Financing  Administration  (HCFA),  in  consultation  with  other  Agencies  in  the  Department  of 
Health  and  Human  Services,  initiated  the  Healthy  Aging  Project  to  enhance  and  promote  the  health  of  older 
people.  A  major  objective  of  the  Healthy  Aging  Project  is  to  identify,  synthesize  and  disseminate  evidence 
and  expert  opinion  on  health  promotion  and  disease  prevention  interventions  that  are  evidence-based.  HCFA 
is  sponsoring  reports  that  present  evidence  and  expert  opinion  to  assist  public  and  private  sector  organizations 
in  their  efforts  to  improve  the  delivery  of  Medicare-covered  preventive  benefits  and  promote  behavioral  risk 
factor  reduction.  These  reports  provide  comprehensive,  science-based  information  on  effective  and  cost- 
effective  interventions  targeting  the  senior  population.  RAND  is  producing  these  reports  under  a  HCFA 
contract. 

HCFA  expects  that  these  evidence  reports  will  inform  peer  review  organizations,  individual  health  plans, 
providers  and  purchasers,  including  Medicare  and  Medicaid,  as  well  as  the  health  care  system  as  a  whole  by 
providing  important  information  to  help  improve  the  delivery  and  quality  of  preventive  health  care  for  older 
people. 

We  welcome  written  comments  on  this  evidence  report.  They  may  be  sent  to: 

Healthy  Aging  Project  Officer 

Division  of  Health  Promotion  and  Disease  Prevention 
Quality  Measurement  and  Health  Assessment  Group 
Office  of  Clinical  Standards  and  Quality 
Health  Care  Financing  Administration 
7500  Security  Blvd. 
MS3-02-01 

Baltimore.  MD  21244-1850 


Jeffrey  Kang,  MD,  MPH 

Chief  Clinical  Officer 

Health  Care  Financing  Administration 


The  Southern  California  Evidence-Based  Practice  Center 


The  Southern  California  Evidence-Based  Practice  Center  is  part  of  the  Evidence-Based  Practice  Program 
sponsored  by  the  Agency  for  Health  Care  Policy  and  Research.  One  of  12  such  Centers  nationwide,  the 
Center  conducts  systematic  reviews  and  technology  assessments  of  all  aspects  of  health  care;  performs 
research  on  improving  the  methods  of  synthesizing  the  scientific  evidence  and  developing  evidence  reports 
and  technology  assessments;  and  provides  technical  assistance  to  other  organizations  in  their  efforts  to 
translate  evidence  reports  and  technology  assessments  into  guidelines,  performance  measures,  and  other 
quality-improvement  tools. 

The  Center  combines  the  talents  of  RAND  and  its  five  affiliated  regional  health  care  institutions:  the 
University  of  California,  Los  Angeles;  the  University  of  California,  San  Diego;  Cedars-Sinai  Medical  Center; 
the  University  of  Southern  California;  and  Value  Health  Sciences.  In  addition,  through  the  VA/RAND/ 
University  of  California  Field  Program  "Center  for  the  Study  of  Health  Care  Provider  Behavior,"  four 
Department  of  Veterans  Affairs  facilities  collaborate  with  the  Center.  The  Center  is  also  affiliated  with  five 
health  services  research  training  programs. 

The  Southern  California  Center  is  the  natural  outcome  of  more  than  20  years  of  work  by  RAND  and  its 
affiliated  institutions  in  reviewing  the  biomedical  literature  for  evidence  of  benefits,  harms,  and  costs;  using 
meta-analysis,  decision  analysis,  and  cost-effectiveness  analysis  to  synthesize  the  literature;  developing 
measures  of  clinical  appropriateness  and  practice  guidelines;  developing  and  assessing  medical  review 
criteria;  and  developing  and  assessing  performance  measures  and  other  tools  for  translating  evidence-based 
knowledge  into  clinical  practice.  The  hallmark  of  this  work  has  been  (a)  its  multi-disciplinary  nature:  RAND 
and  its  affiliated  institutions  combine  the  talents  of  clinicians,  health  services  researchers,  epidemiologists, 
statisticians,  economists,  and  advanced  methods  experts  in  meta-analysis  and  decision  analysis;  (b)  the 
advancement  of  knowledge  about  the  methods  for  performing  literature  reviews,  synthesizing  evidence,  and 
developing  practice  guidelines  or  review  criteria;  and  (c)  the  emphasis  on  developing  and  evaluating  products 
for  use  in  the  real  world  of  health  care  delivery. 
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EXECUTIVE  SUMMARY 
AND  EVIDENCE-BASED  RECOMMENDATIONS 

The  proportion  of  the  U.S.  population  over  age  65  has  increased  from  5%  in  1900  to  13%  in  1997.  This 
change  in  demographics,  combined  with  an  increase  in  average  life  expectancy,  has  highlighted  the 
importance  of  preventive  care  services  for  older  individuals.  For  example,  in  1993,  the  five  most  common 
causes  of  mortality  among  patients  age  65  and  over  were  heart  disease,  cancer,  cerebrovascular  diseases, 
chronic  obstructive  pulmonary  disease,  pneumonia,  and  influenza,  all  of  which  are  potentially  preventable. 
Early  detection  and  treatment  can  improve  the  health  of  patients  with  these  and  many  other  conditions  and 
potentially  decrease  health  care  costs.  Similarly,  appropriate  immunizations  can  prevent  some  of  these 
conditions  from  occurring,  or  render  the  disease  less  severe.  To  determine  the  best  strategies  for  early 
detection  and  prevention  currently  covered  by  Medicare  for  several  of  these  conditions,  the  Health  Care 
Financing  Administration  (HCFA)  has  commissioned  this  project  to  assess  interventions  designed  to  improve 
influenza  and  pneumococcal  immunization  rates,  mammography  rates,  cervical  smear  cytology  (pap  test) 
rates,  and  colon  cancer  screening  rates. 

METHODS 

In  this  report,  we  synthesize  evidence  from  the  scientific  literature  using  the  methods  of  the  Southern 
California  Evidence  Based  Practice  Center  (EPC).  an  Agency  for  Health  Care  Policy  and  Research-designated 
center  for  the  systematic  review  of  literature  on  the  evidence  of  the  benefits  and  harms  of  health  care 
interventions.  Our  literature  review  process  utilized  the  following  steps: 

•  develop  a  conceptual  model 

•  identify  sources  of  evidence 

•  identify  potential  evidence 

•  evaluate  potential  evidence  for  methodologic  quality  and  relevance 

•  extract  study-level  variables  and  results  from  studies  meeting  methodologic  and  clinical  criteria 

•  synthesize  the  results. 
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For  this  study,  HCFA  specified  that  we  review  the  evidence  on  the  five  screening/vaccination  services 
currently  covered  by  Medicare  (i.e.,  influenza  and  pneumonia  vaccinations,  screening  mammography, 
cervical  smear  cytology,  colon  cancer  screening),  using  a  target  population  of  persons  over  age  65.  HCFA 
specified  the  relevant  outcome  as  receipt  of  these  items  by  the  target  population. 

Our  conceptual  model  classified  the  potential  targets  of  intervention  as  patient,  provider,  organization,  or 
community.  It  classified  the  types  of  interventions  as  either  reminders,  feedback,  education,  financial 
incentives,  regulatory  and  legislative  interventions,  organizational  change,  or  media  campaigns.  These 
intervention  types  are  defined  in  the  Glossary  of  Terms  located  in  the  Appendix. 

We  used  the  following  five  sources  to  identify  existing  research  and  potentially  relevant  evidence  for  this 
report:  Cochrane  Effective  Practice  and  Organization  of  Care  (EPOC)  Data  Base,  two  previous  systematic 
reviews,  a  draft  chapter  on  immunizations  from  the  Centers  for  Disease  Control  and  Prevention  (CDC)  and 
Health  Care  Quality  Improvement  Projects  (HCQIP)  done  by  the  Medicare  Peer  Review  Organizations 
(PROs).  Documents  describing  the  HCQIPs  are  called  Narrative  Project  Descriptions  (NPDs). 

After  retrieving  articles  from  the  five  sources,  we  reviewed  them  against  exclusion  criteria.  To  be  accepted  at 
this  stage,  a  study  had  to  address  one  or  more  of  the  five  services  of  interest  and  employ  one  of  the  following 
study  designs:  randomized  controlled  trial,  controlled  clinical  trial,  controlled  before  and  after  study,  or 
interrupted  time  series.  While  we  were  primarily  searching  for  data  relevant  to  the  Medicare  population,  we 
included  studies  that  reported  data  on  populations  under  age  65  at  this  stage  to  avoid  premature  loss  of 
potentially  useful  data.  Because  of  the  restrictions  on  study  design,  we  excluded  studies  that  employed  a 
simple  pre/post  design.  Such  a  study  design  has  no  control  group;  therefore,  it  cannot  account  for  temporal 
effects  unrelated  to  the  intervention. 

After  retrieving  the  relevant  articles,  we  abstracted  data  about  the  study  design;  the  number  and  characteristics 
of  patients;  the  setting,  location,  and  target  of  the  intervention;  the  intensity  of  the  intervention;  the  types  of 
outcome  measures;  the  time  from  intervention  until  outcome  measurement;  and  the  results.  Two  physicians 
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working  independently  extracted  data  in  duplicate,  compared  data  and  resolved  discrepancies  by  consensus. 
A  senior  project  member  resolved  discrepancies  not  resolvable  by  consensus. 

ANALYSIS 

All  analyses  were  performed  using  the  statistical  analysis  package  SAS.  We  calculated  the  risk  difference 
between  groups  for  each  comparison  in  each  study.  The  Number  Needed  to  Treat  (NNT)  was  then  calculated 
as  the  inverse  of  the  risk  difference. 

We  conducted  a  meta-regression  analysis  to  determine 

•  the  absolute  and  relative  effectiveness  of  each  different  intervention  component,  such  as  patient 
reminders,  adjusted  for  other  intervention  components,  and  controlling  for  measured  and  unmeasured 
study  differences; 

•  the  effect  of  each  important  covariate,  such  as  target  population,  on  the  effectiveness  of  interventions 
controlling  for  measured  and  unmeasured  study  differences;  and 

•  whether  an  intervention  factor,  such  as  top  management  buy-in  and  support,  was  instrumental  in  the 
success  of  interventions  controlling  for  other  intervention  factors,  and  measured  and  unmeasured  study 
differences. 

These  multivariate  models  produced  an  adjusted  estimate  of  the  odds  ratio  for  receiving  a  screening  service  if 
one  is  subject  to  an  intervention  containing  a  particular  intervention  component,  to  the  odds  if  one  belongs  to 
the  control  or  usual  care  group.  These  models  also  produce  an  adjusted  marginal  odds  ratio  that  represents  the 
change  in  the  odds  of  receiving  the  screening  service  if  a  particular  intervention  component  is  added  to  the 
"average"  intervention  without  that  component.  Similar  statistics  are  available  for  the  covariate  and 
intervention  factor  models. 

DEVELOPMENT  OF  RECOMMENDATIONS 

Based  on  the  evidence,  we  developed  several  recommendations  regarding  interventions  to  increase  the  use  of 

preventive  and  screening  services  in  the  Medicare  population.  These  recommendations  were  distributed  to 

over  a  dozen  experts  in  the  fields  of  public  health,  preventive  medicine  and  geriatrics  for  their  review  and 

6 


feedback.  Expert  reviewers  are  listed  in  Table  16  in  the  Appendix.  The  recommendations  made  in  this 
document  incorporate  many  suggestions  from  these  experts. 

RESULTS 

A  total  of  187  articles,  presenting  data  on  218  unique  studies,  passed  our  screening  criteria.  Of  these,  136 
were  randomized  clinical  trials,  24  were  controlled  clinical  trials,  and  58  were  controlled  before-and-after 
studies.  We  found  no  time  series  studies  meeting  our  inclusion  criteria  (at  least  three  measurement  points 
both  before  and  after  implementation  of  the  intervention).  In  the  2 1 8  studies,  there  were  288  comparisons  of 
an  intervention  to  usual  care  or  a  control  group.  These  were  distributed  as  follows: 

•  78  controlled  comparisons  of  interventions  to  increase  influenza  immunization 

•  18  controlled  comparisons  to  increase  the  use  of  pneumococcal  immunization 

•  76  controlled  comparisons  of  interventions  to  increase  the  use  of  mammography 

•  65  controlled  comparisons  to  increase  the  use  of  cervical  smear  cytology 

•  41  controlled  comparisons  to  increase  fecal  occult  blood  testing  for  colon  cancer 

•  10  controlled  comparisons  to  increase  the  use  of  methods  for  colon  visualization. 

We  used  meta-regression  to  summarize  these  studies,  combining  the  immunization  studies  into  a  single 
analysis  and  treating  each  of  the  cancer  screening  tests  separately.  There  were  too  few  studies  to  do  an 
analysis  on  interventions  to  increase  the  use  of  colon  visualization  (i.e..  sigmoidoscopy,  colonoscopy,  or 
barium  enema).  Not  all  interventions  had  sufficient  information  for  all  preventive  services;  at  least  two 
separate  studies  were  needed  in  order  to  enter  any  intervention  into  the  model. 

Across  all  four  regressions,  there  were  some  consistent  patterns.  First,  organizational  change  was  consistently 
one  of  the  most  or  the  most  effective  interventions  at  increasing  use  of  these  clinical  and  preventive  services. 
Second,  patient  financial  incentives  were  also  highly  effective  at  increasing  the  use  of  the  all  services.  Third, 
patient  reminders  demonstrated  a  relatively  consistent  effect  across  all  services,  as  did  patient  education.  For 
patient  reminders,  there  is  some  evidence  that  personalized  reminders  (or  those  signed  by  the  patient's 
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physician)  are  more  effective  than  generic  reminders.  Finally,  feedback  appeared  to  be  a  relatively  ineffective 
intervention,  as  it  was  statistically  beneficial  only  for  increasing  mammography  screening. 

The  sample  sizes  involved  in  studies  of  mass  mailings  for  influenza  vaccine  made  them  unfeasible  to  enter 
into  the  statistical  model;  these  studies  were  summarized  separately.  There  was  one  study  from  the  Narrative 
Project  Description  file  that  reported  a  statistically  significant  benefit  and  was  subsequently  published  in 
Morbidity  and  Mortality  Weekly  Report  (1995).  Four  studies  that  did  not  report  a  clinically  meaningful 
benefit  were  found  in  the  Narrative  Project  Description  files. 

In  addition  to  these  main  results,  we  performed  other  analyses  to  test  possible  relationships  between  efficacy 
of  an  intervention  and  baseline  immunization  rate,  intensity  of  intervention,  and  year  of  publication.  Our 
analyses  were  limited  by  the  large  amount  of  missing  data  (i.e.,  studies  did  not  report  a  baseline  rate  or  did  not 
report  information  about  the  intensity  of  the  intervention).  Our  results  were  not  consistent,  and  the 
relationship  between  efficacy  and  any  of  these  other  variables,  based  on  these  data,  is  not  strong. 

LIMITATIONS 

The  primary  limitation  of  this  review  is  the  quantity  and  quality  of  the  original  studies.  Even  more  so  than 
reviews  of  single  therapies  for  single  conditions,  the  studies  presented  here  are  extremely  heterogeneous  in 
terms  of  the  interventions  that  were  tested  and  the  specific  populations  or  health  care  systems  being  studied. 
An  additional  limitation  is  that  many  of  the  original  studies  did  not  include  sufficient  detail  to  allow  statistical 
testing.  Lastly,  since  our  data  were  abstracted  from  articles  that  studied  both  under  65  year  old  as  well  as  the 
over  65  population,  it  is  possible  that  our  conclusions  are  not  entirely  applicable  to  the  Medicare  population. 
Lastly,  for  several  services  there  were  very  few  studies  upon  which  to  base  conclusions. 
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CONCLUSIONS 

Keeping  in  mind  the  limitations  noted  above,  and  that  knowledge  of  local  barriers  and  opportunities  to 
improve  services  is  a  key  ingredient  in  designing  effective  interventions,  we  draw  the  following  conclusions 
from  the  literature: 

1 .  Organizational  change  (such  as  standing  orders)  and  financial  incentives  are  the  interventions  that  were 
most  consistent  at  producing  the  largest  improvements  in  use  of  all  preventive  and  screening  services. 

2.  Patient  reminders  are  also  consistently  effective  across  all  preventive  and  screening  services,  although  in 
general,  they  were  less  so  than  organizational  change  or  financial  incentives.  Patient  reminders  that  are 
personalized  or  signed  by  the  patient's  physician  are  more  effective  than  reminders  that  are  generic. 

3.  Provider  reminders  are  very  effective  at  improving  receipt  of  immunizations  and  show  consistent  but 
moderate  effectiveness  at  improving  the  use  of  cancer  screening  services. 

4.  Patient  education  is  consistently  less  effective  than  organizational  change  and  reminders.  The  effect  of 
patient  education,  while  still  significant,  is  modest. 

5.  Feedback  is  of  limited,  if  any,  effectiveness. 

6.  Mass  mailings  by  PROs  to  improve  influenza  immunizations  have  been  shown  to  produce  clinically 
trivial  effects  when  unaccompanied  by  other  interventions. 

7.  Multiple  interventions  are  more  effective  than  single  interventions,  although  highly  successful  single 
interventions  exist.  Adding  organizational  change  or  reminders  to  an  "average"  intervention  produces  the 
greatest  increase  in  effectiveness.  However,  the  relative  cost  effectiveness  of  adding  interventions  has  not 
been  established. 

8.  Computer-assisted  provider  reminders  are  more  cost-effective  than  patient  reminders  in  the  few  studies 
that  have  addressed  this  issue. 


9.  There  are  insufficient  data  to  draw  conclusions  about  which  interventions  are  most  effective  for  special 
populations,  geographic  settings,  or  delivery  systems. 

10.  There  are  insufficient  data  to  draw  conclusions  about  the  effect  of  pre-intervention  rates,  intensity  of 
interventions,  or  other  factors  in  determining  the  success  of  interventions. 
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RECOMMENDATIONS 

1.  No  intervention  is  consistently  effective  for  all  services  and  all  settings  (i.e.,  there  is  no  "magic  bullet"). 
To  be  most  effective,  interventions  should  address  the  barriers  to  obtaining  clinical  preventive  services 
that  are  specific  to  the  service  and  the  local  setting,  and  they  should  take  into  account  the  cost  and 
difficulty  of  implementation. 

2.  Many  interventions  increase  the  receipt  of  immunizations:  organizational  change,  provider  and  patient 
reminders,  provider  and  patient  financial  incentives,  and  provider  and  patient  education.  Although  all 
these  interventions  are  effective  at  improving  immunization  rates,  some  are  more  successful  than  others. 
Therefore,  the  subsequent  recommendations  reflect  these  differences: 

•  Standing  orders  (which  we  classify  as  a  type  of  organizational  change)  should  be  implemented  in  any 
clinical  setting  in  which  they  are  applicable,  such  as  hospitals,  nursing  homes,  and  clinics.  This 
intervention  is  likely  to  be  both  low  cost  and  very  effective. 

•  Patient  reminders  or  provider  reminders  should  be  implemented  in  any  setting  in  which  standing 
orders  cannot  be  implemented.  Patient  reminders  that  are  personalized  or  signed  by  the  patient's 
doctor  appear  to  be  more  effective  than  generic  reminders.  Patient  reminders  have  a  low  estimated 
cost  ($1  to  $5)  per  unit,  while  the  cost  of  provider  reminders  is  likely  to  be  very  dependent  upon  the 
state  of  the  available  information  technology  system.  When  a  computerized  information  technology 
system  is  available,  computerized  provider  reminders  have  been  consistently  more  cost-effective  than 
patient  reminders. 

•  Mass  mailings  of  post  cards,  letters,  and  brochures  by  Peer  Review  Organizations  have  been  shown  to 
produce  clinically  trivial  effects  on  influenza  immunization  rates  when  unaccompanied  by  other 
interventions.  Therefore,  this  approach  should  be  abandoned. 
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3.   Many  interventions  improve  rates  for  cancer  screening.  Some  interventions  are  consistently  more 
effective  than  others,  and  the  subsequent  recommendations  reflect  these  differences: 

•  Organizational  change  is  consistently  one  of  the  most  effective  interventions  for  improving  cancer 
screening,  and  should  be  implemented  whenever  possible.  However,  there  is  insufficient  evidence  to 
recommend  as  superior  any  particular  organizational  change.  Examples  of  successful  organizational 
change  include: 

The  use  of  a  "prevention  team"  that  included  nurse  standing  orders  for  mammography 

The  use  of  nurses  to  distribute  kits  and  instruct  patients  in  the  completion  of  fecal  occult  blood 
testing 

The  use  of  a  health  educator  to  contact  patients  via  telephone  to  offer  barrier  counseling  and/or 
assist  in  appointment  scheduling  for  cervical  cancer  screening. 

•  Patient  reminders  and  provider  reminders  are  effective  at  increasing  the  use  of  cancer  screening  and 
either  one  should  be  implemented  in  any  setting  in  which  organizational  change  cannot  be 
implemented.  Patient  reminders  that  are  personalized  or  signed  by  the  patient's  doctor  appear  to  be 
modestly  more  effective  than  generic  reminders.  Patient  reminders  have  a  low  estimated  cost  ($1  to 
$5)  per  unit,  while  the  cost  of  provider  reminders  is  likely  to  be  very  dependent  upon  the  state  of  the 
available  information  technology  system.  When  a  computerized  information  technology  system  is 
available,  computerized  provider  reminders  have  been  consistently  more  cost-effective  than  patient 
reminders. 

•  Provider  education  is  moderately  effective  at  increasing  the  use  of  cancer  screening,  and  it  should  be 
considered  in  any  situation  in  which  organizational  change  and/or  reminders  cannot  be  implemented. 
Provider  education  receives  a  lower  recommendation  than  patient  reminders  or  provider  reminders 
based  both  on  the  evidence  of  its  lower  efficacy  and  its  likely  expense. 
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4.  In  general,  patient  education  and  provider  feedback  are  not  as  effective  as  other  interventions  for  most 
screening  and  preventive  services;  therefore,  they  should  not  be  the  first  choice  for  intervention. 

5.  Multiple  interventions  should,  on  average,  produce  greater  effectiveness  than  single  interventions, 
although  the  incremental  cost-effectiveness  of  adding  interventions  remains  unknown. 
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EVIDENCE  REPORT 
INTRODUCTION 

The  proportion  of  the  U.S.  population  over  age  65  has  increased  from  5%  in  1900  to  13%  in  1997.  This 
change  in  demographics,  combined  with  an  increase  in  average  life  expectancy,  has  highlighted  the 
importance  of  preventive  care  services  for  older  individuals  (Rowe,  1999).  For  example,  in  1993,  the  five 
most  common  causes  of  mortality  among  patients  age  65  and  over  were  heart  disease,  cancer,  cerebrovascular 
diseases,  chronic  obstructive  pulmonary  disease,  and  pneumonia  and  influenza,  all  of  which  are  potentially 
preventable  (CDC,  1996).  Early  detection  and  treatment  can  improve  the  health  of  patients  with  many  of 
these  conditions  and  potentially  decrease  health  care  costs.  Similarly,  appropriate  immunizations  can  prevent 
several  of  these  conditions  from  occurring,  or  render  the  disease  less  severe.  To  help  determine  the  best 
strategies  for  early  detection  and  prevention  of  some  of  these  conditions,  the  Health  Care  Financing 
Administration  (HCFA)  has  commissioned  this  project  to  assess  interventions  designed  to  improve  influenza 
and  pneumococcal  immunization  rates,  mammography  rates,  cervical  smear  cytology  rates,  and  colon  cancer 
screening  rates. 

Influenza  Immunization 

Influenza  and  consequent  respiratory  diseases  are  common  causes  of  morbidity  and  mortality  in  the  United 
States  each  year,  with  20,000  to  40.000  deaths  reported  for  each  influenza  epidemic.  Over  90%  of  these 
deaths  occur  among  those  age  65  or  older  (Centers  for  Disease  Control  and  Prevention,  1996).  Influenza 
vaccination  has  been  shown  to  be  efficacious  in  the  elderly,  decreasing  hospitalizations  by  27%  to  57% 
(Nichol,  1994)  and  deaths  by  27%  to  30%  (Fedson,  1993). 

Pneumococcal  Vaccination 

Pneumococcal  infection  is  common  in  the  United  States,  accounting  for  15%  of  severe  community-acquired 
pneumonia  (US  Preventive  Services  Task  Force,  1996). 

14 


Invasive  pneumococcal  infection  has  a  mortality  rate  of  over  30%  in  elders  (Haglund,  1993;  Wenger,1990; 
Breiman,  1990;  Bennett,  1992;  Jette,  1989).  Case  control  and  epidemiological  studies  have  shown  that 
pneumococcal  vaccination  has  an  aggregate  efficacy  of  approximately  55%  to  70%  for  preventing 
pneumococcal  infection  in  elderly  persons  (American  College  of  Physicians,  1994).  The  newer  23-valent 
vaccine  may  prove  to  be  even  more  effective  in  this  population. 

Mammography 

Approximately  one-third  of  newly  diagnosed  cancers  in  American  women  consists  of  cancers  of  the  breast. 
In  1995  alone,  there  were  182,000  new  breast  cancer  cases,  and  46,000  deaths  attributable  to  breast  cancer 
(Wingo,  1995). 

Slightly  less  than  half  of  all  new  breast  cancer  cases,  but  slightly  more  than  half  of  breast  cancer  deaths,  occur 
among  women  age  65  or  older  (Ries,  1994).  Although  routine  mammography  has  been  shown  to  decrease 
breast  cancer  mortality  among  women  over  age  50  by  20%  to  30%  (U.S.  Preventive  Services  Task  Force, 
1996),  only  slightly  more  than  half  (58%)  of  women  age  55  or  older  have  ever  received  a  mammogram 
(Ruchlin,  1997). 

Three  trials  have  specifically  addressed  the  effectiveness  of  mammography  in  older  women  (i.e.,  those  aged 
70  to  74).  Two  of  them  found  that  mammography  screening  reduced  breast  cancer  mortality  (Tabar,  1992; 
Morrison,  1988).  Although  the  third  study  showed  no  improvement  in  mortality,  only  a  small  number  of 
patients  were  studied,  which  produced  a  large  confidence  interval  (Nystrom,  1993). 

Cervical  Smear  Cytology 

Cervical  smear  cytology  tests  are  associated  with  a  20%  to  60%  decrease  in  cervical  cancer  mortality 
(Cramer,  1974;  Miller,  1976;  Anderson,  1988:  Johanneson.  1978;  Laara,  1987;  Boon,  1990;  Costa,  1991; 
Benedet,  1992;  Sigurdsson,  1993).  Although  recent  improvements  in  cervical  smear  cytology  rates  have 
correlated  with  a  decrease  in  the  incidence  of  advanced  cervical  cancer  in  the  United  States  (U.S.  Preventive 
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Services  Task  Force,  1996),  there  were  still  16,000  new  cervical  cancer  cases  and  4,800  deaths  attributable  to 
cervical  cancer  in  1995  (Wingo,  1995). 

The  U.S.  Preventive  Services  Task  Force  currently  recommends  that  all  women  over  age  65  receive  at  least 
one  cervical  smear  cytology  test  if  they  have  not  received  such  tests  regularly  in  the  past. 

Colon  Cancer  Screening 

Colorectal  cancer  accounts  for  approximately  140.000  new  cases  of  cancer  and  55,000  deaths  each  year, 
making  it  the  second  most  common  cause  of  cancer  death  in  the  United  States  (Wingo,  1995).  The  risk  of 
experiencing  colorectal  cancer  increases  with  age. 

According  to  Mandel  and  colleagues  (1993),  mortality  from  colorectal  cancer  decreases  by  about  one-third  if 
lesions  are  detected  early  and  treatment  is  initiated.  Both  fecal  occult  blood  testing  and  sigmoidoscopy  have 
been  shown  to  be  effective  in  colorectal  cancer  screening  (U.S.  Preventive  Services  Task  Force,  1996); 
therefore,  the  U.S.  Preventive  Services  Task  Force  recommends  that  all  individuals  over  age  65  undergo 
either  fecal  occult  blood  testing  or  sigmoidoscopy  at  least  once. 
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METHODS 

In  this  report,  we  synthesize  evidence  from  the  scientific  literature  on  improving  the  utilization  of  existing 
Medicare  clinical  preventive  and  screening  services  among  persons  age  65  or  older.  The  services  under 
review  are  mammography  for  breast  cancer  screening  in  women,  cervical  smear  cytology  for  cervical  cancer 
screening,  fecal  occult  blood  or  colon  visualization  for  colon  cancer  screening,  influenza  vaccination,  and 
pneumococcal  vaccination. 

We  employed  the  evidence  review  and  synthesis  methods  of  the  Southern  California  Evidence  Based  Practice 
Center,  an  Agency  for  Health  Care  Policy  and  Research-designated  center  for  the  systematic  review  of 
literature  on  the  evidence  for  benefits  and  harms  of  health  care  interventions.  Our  literature  review  process 
utilized  the  following  steps: 

•  develop  a  conceptual  model  (also  sometimes  called  an  evidence  model  or  a  causal  pathway  (Woolf, 
1994)) 

•  identify  sources  of  evidence  (in  this  case,  sources  of  scientific  literature) 

•  identify  potential  evidence 

•  evaluate  potential  evidence  for  methodologic  quality  and  relevance 

•  extract  study-level  variables  and  results  from  studies  meeting  methodologic  and  clinical  criteria 

•  synthesize  the  results. 

We  present  descriptions  of  each  step  below. 
Development  of  Conceptual  Model 

We  initially  met  with  the  project's  technical  experts  on  behavior  change  (Jeremy  Grimshaw,  M.D.,  from  the 
Health  Services  Research  Unit  of  the  University  of  Aberdeen  and  the  coordinating  editor  of  the  Cochrane 
Effective  Practice  and  Organization  of  Care  Group;  Brian  Mittman,  Ph.D.  and  Lisa  Rubenstein,  M.D., 
M.S.P.H,  both  from  the  Department  of  Veterans  Affairs  Center  for  the  Study  of  Health  Care  Provider 
Behavior)  to  discuss  which  interventions  and  study-level  variables  to  extract  from  the  literature.  Subsequent 
discussions  between  project  staff  and  HCFA  concluded  that  we  should  review  the  evidence  on  the  five 

17 


screening/vaccination  items  currently  covered  by  Medicare  (e.g.,  influenza  and  pneumonia  vaccinations, 
mammography,  cervical  smear  cytology,  and  colon  cancer  screening),  using  a  target  population  of  persons 
over  age  65.  These  discussions  further  specified  the  relevant  outcomes  as  receipt  of  these  items  by  the  target 
population  (i.e.,  the  use  of  screening  mammography  and  the  other  items  was  assumed  a  priori  to  be 
efficacious  in  producing  better  health  outcomes,  thereby  justifying  receipt  of  the  item  as  a  valid  measure  of 
outcome). 

Using  these  parameters  and  input  from  the  technical  experts,  we  developed  the  project's  conceptual  model 
from  a  larger  model  produced  by  the  Veterans  Administration  (VA)  Center  for  the  Study  of  Health  Care- 
Provider  Behavior  Change  (Table  1). 
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Center  for  the  Study  of 
Healthcare  Provider  Behavior 


Table  1.  Classification  for  Interventions  That  Change  Behavior 
Developed  by  VA  Center  for  the  Study  of  Provider  Behavior 


A.  Passive  dissemination  of  recommendation 


1 .  Published  clinical  practice  guideline 

2.  Published  clinical  recommendation 

3.  Published  research  finding 

4.  Postal  mailed  clinical  practice  guideline 

5.  Postal  mailed  educational  or  technical  bulletin 

6.  Postal  mailed  research  finding  or  alert 

7.  Internal  (organizational)  mailed  clinical  practice  guideline 

8.  Internal  (organizational)  mailed  educational  or  technical  bulleting 

9.  Internal  (organizational  mailed)  research  finding  or  alert 

10.  Internal  (organizational  mailed)  organizational  policy  statement  [but  see  administrative  interventions] 

11.  Clinical  practice  guideline  distributed  at  an  internal  (organizational)  meeting 

12.  Organizational/departmental  policy  statement  distributed  at  an  internal  (organizational)  meeting 


B.  Education,  Opinion  Leaders 

1.  One-session  educational  lecture,  seminar,  talk  (with  or  without  educational  handouts),  provided  at  a 
national  or  regional  professional  meeting;  attendees  might  include  known  local  peers  but  majority  are 
not  local  peers 

2.  One-session  educational  lecture,  seminar,  talk  (with  or  without  educational  handouts),  provided  at  a 
local  professional  meeting;  attendees  are  known  local  peers 

3.  One-session  educational  lecture,  seminar,  talk  (with  or  without  educational  handouts),  provided  at  a 
university  or  other  independent  institution;  attendees  might  include  known  local  peers 

4.  One-session  educational  lecture,  seminar,  talk  (with  or  without  educational  handouts),  provided 
within  a  provider  organization  (hospital,  medical  group) 

5.  Two-session  educational  lecture,  seminar,  talk,  provided  at  a  university  or  other  independent 
institution 

6.  Two-session  educational  lecture,  seminar,  talk,  provided  within  a  provider  organization  (hospital, 
medical  group) 

7.  Three-i-  session  educational  lecture,  seminar,  talk,  provided  at  a  university  or  other  independent 
institution 

8.  Three+  session  educational  lecture,  seminar,  talk,  provided  within  a  provider  organization  (hospital, 
medical  group) 


C.  Audit/feedback  without  opinion  leader/study  group  social  influence  component 

1 .  Mailed  distribution  of  aggregate-level  performance  data 

2.  Mailed  distribution  of  individual-level  performance  data 

3.  Group-setting  distribution  of  aggregate-level  performance  data,  without  discussion 
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Table  1.  Classification  for  Interventions  That  Change  Behavior 
Developed  by  VA  Center  for  the  Study  of  Provider  Behavior 

(continued) 

4.  Group-setting  distribution  of  individual-level  performance  data,  with  peer  comparison,  without 
discussion 

5.  Group-setting  distribution  of  individual-level  performance  data,  without  peer  comparison,  without 
discussion 

Audit/feedback  with  opinion  leader/study  group  social  influence  component 

1 .  Group-setting  distribution  of  aggregate-level  performance  data,  with  opinion  leader 
lecture/presentation  discussion 

2.  Group-setting  distribution  of  aggregate-level  performance  data,  with  group  (peer)  discussion  (with  or 
without  opinion  leader  lecture/presentation  discussion) 

3.  Group-setting  distribution  of  individual-level  performance  data  without  peer  comparison,  with 
opinion  leader  lecture/presentation  discussion 

4.  Group-setting  distribution  of  individual-level  performance  data  with  peer  comparison,  with  opinion 
leader  lecture/presentation  discussion 

5.  Group-setting  distribution  of  individual-level  performance  data  without  peer  comparison,  with  group 
(peer)  discussion  (with  or  without  opinion  leader  lecture/presentation  discussion) 

6.  Group-setting  distribution  of  individual-level  performance  data  with  peer  comparison,  with  group 
(peer)  discussion  (with  or  without  opinion  leader  lecture/presentation  discussion) 

7.  Individual-setting  provision  of  individual-level  performance  data  without  peer  comparison 

8.  Individual-setting  provision  of  individual-level  performance  data  with  peer  comparison 

Educational  visits/detailing/outreach 

1 .  Detailing  visits  provided  by  a  known  local  peer  of  like  discipline 

2.  Detailing  visits  provided  by  a  known  local  individual  in  a  different  discipline 

3.  Detailing  visits  provided  by  a  non-local  individual  of  like  discipline 

4.  Detailing  visits  provided  by  a  non-local  individual  in  a  different  discipline 

Physician  reminders 

1.  Hard-copy  reminders,  sent  via  mail  or  other  general  distribution 

2.  Hard-copy  reminders,  placed  in  charts 

3.  Hard-copy  reminders,  provided  at  time  of  patient  visit,  other  than  in  charts 

4.  Computerized  reminders,  sent  via  email-general  distribution 

5.  Computerized  reminders,  provided  by  electronic  charting,  other  point-of-care/patient  bedside  manner 


20 


Table  1.  Classification  for  Interventions  That  Change  Behavior 
Developed  by  VA  Center  for  the  Study  of  Provider  Behavior 

(continued) 

G.  Organizational  quality  improvement,  redesign  (reengineering) 

1.  Total  Quality  Management/  Continuos  Quality  Improvement  (TQM/CQI)  intervention  involving 
interdisciplinary  quality  team  operating  under  an  organizational  quality  council  with  full  TQM/CQI 
process  (diagnosis  and  process-mapping,  data  collection/analysis,  solution  formulation, 
implementation) 

2.  TQM/CQI  intervention  with  less  than  full  (conventional)  process 

3.  Other  system/policy/procedure  change  (e.g.,  patient  visit  protocol  including  automatic 
screening/vaccination  administered  by  nurse  during  vitals/pre-physician  encounter  phase  of  visit) 


H.  Administrative  interventions,  financial  incentives 

1.  Promulgation  of  departmental  policy  without  monitoring  or  incentives/sanctions 

2.  Promulgation  of  departmental  policy  with  performance  monitoring  but  no  incentives/sanctions 

3.  Promulgation  of  departmental  policy  with  performance  monitoring  and  no  (explicit) 
incentives/sanctions 

4.  Promulgation  of  departmental  policy  with  performance  monitoring  and  incentives  (rewards)  for  good 
performance 

5.  Promulgation  of  departmental  policy  with  performance  monitoring  and  sanctions  for  poor 
performance 

6.  Promulgation  of  departmental  policy  with  performance  monitoring  and  incentives  and  sanctions 

7.  Payor-provided  incentives 


I.  Patient  reminders,  incentives 

1 .  Hard-copy  reminders,  sent  via  mail  or  other  general  distribution 

2.  Patient  financial  incentives  (reduced/eliminated  co-pay;  gift) 


J.  Mass  media,  other  community-level  interventions  (outside  the  healthcare  provider  organization) 


1.  Mass  media  publication  targeting  patients  (and  providers?) 

2.  Community-based  screening/vaccination  programs/events 
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Under  the  advice  of  the  technical  experts,  we  reduced  the  VA  model's  full  classification  scheme  down  to 
seven  broad  categories  of  interventions,  or  key  domains,  that  could  be  used  to  increase  utilization  of 
mammograms,  cervical  smear  cytology,  colon  cancer  screening,  pneumococcal  vaccination,  and  influenza 
vaccination  among  persons  age  65  or  older: 

•  reminders 

•  feedback 

•  education 

•  financial  incentives 

•  regulatory  and  legislative  interventions 

•  organizational  change 

•  media  campaigns. 

In  addition,  we  categorized  the  four  potential  targets  of  intervention  as  patient,  provider,  organization,  and 
community. 

The  final  conceptual  model  is  depicted  in  Figure  1 .  A  detailed  description  of  each  intervention  category  is 
presented  below. 

Reminders.  These  can  be  provided  verbally,  on  paper,  or  on  a  computer  screen.  They  can  be  intended  to 
prompt  a  health  professional  to  recall  information  or  a  patient  to  utilize  a  service.  They  can  be  mailed  to 
patients,  placed  in  charts,  or  even  sent  via  e-mail  to  providers. 

Feedback.  Feedback  occurs  when  a  summary  of  clinical  performance  over  a  specified  period  of  time  is  given 
to  a  provider.  The  summary  may  include  recommendations  for  clinical  action.  The  information  may  have 
been  obtained  from  medical  records,  computerized  databases,  or  observations  from  patients.  Audit  and 
feedback  are  often  combined  with  opinion  leader  counseling  of  clinicians,  and  have  the  potential  both  to 
improve  clinical  knowledge  and  to  create  social  influences  that  may  improve  performance. 
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Education.  Medical  education  interventions  span  a  broad  array  of  methods  for  disseminating  information  to 
health  professionals,  including  distribution  of  published  or  printed  recommendations  for  clinical  care,  such  as 
clinical  practice  guidelines,  audio-visual  materials,  and  electronic  publications.  Materials  may  be  delivered 
personally  or  through  mass  mailings.  Providers  may  also  attend  conferences,  workshops,  trainings,  or 
lectures.  Patients  may  be  educated  through  pamphlets,  peer  educators,  newsletters,  and  so  forth. 

Financial  Incentives.  Direct  or  indirect  financial  reward  or  benefit  can  be  tied  to  a  specific  action  on  the  part 
of  a  provider  or  a  patient.  For  example,  patient  co-payments  may  be  eliminated  or  reduced,  or  gifts  can  be 
offered  to  patients  as  incentives. 

Regulatory  &  Legislative  Interventions.  These  initiatives  are  not  often  viewed  as  healthcare  provider 
behavior  change  interventions  because  they  operate  at  a  national  or  state  level,  but  such  initiatives  can  have 
significant,  if  sometimes  indirect,  influences  on  provider  behavior.  Legislative  and  regulatory  initiatives 
operate  by  changing  the  environment  and  organizations  within  which  providers  practice,  and  by  creating  new 
incentives  and  barriers  that  shape  behavior.  Examples  include  changes  in  medical  liability,  licensure 
requirements,  and  management  of  patient  complaints. 

Organizational  Change.  We  defined  organizational  change  as  any  change  in  the  process  of  care  at  a  clinic, 
program,  or  hospital  specifically  designed  to  improve  preventive  care  services.  This  definition  encompasses 
adding  new  personnel  with  new  functions  to  the  process  of  care,  changing  the  process  by  which  patients 
obtain  services  or  appointments,  changing  the  role  of  a  current  provider,  or  improving  the  facilities  or 
infrastructure.  Additional  examples  of  organizational  change  include  adding  case  management  (e.g., 
coordination  of  assessment,  treatment,  and  arrangement  for  referrals)  and  revision  of  professional  roles,  such 
as  shifting  roles  among  health  care  professionals  (i.e.,  from  doctors  to  nurses,  psychologists  to  social 
workers). 
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Media  Campaigns.  Media  campaigns  use  communication  that  reaches  great  numbers  of  people  including 
television,  radio,  newspapers,  posters,  leaflets  and  booklets,  alone  or  in  conjunction  with  other  interventions. 
Campaigns  are  usually  targeted  at  the  population  level. 

Delivery  strategies,  such  as  those  listed  at  the  bottom  of  Figure  1,  can  significantly  affect  the  success  of  these 
interventions.  Most  of  these  service  delivery  elements  incorporate  widely  accepted  psychological  theory.  For 
example,  organizational  interventions  should  have  the  full  support  of  upper  management,  and  involve 
teamwork  and  collaboration  between  staff  levels;  media  campaigns  should  have  high  visual  appeal;  and 
educational  sessions  should  use  active  learning  strategies. 
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Identification  of  Literature  Sources 

We  used  the  five  sources  described  below  to  identify  existing  research  and  potentially  relevant  evidence  for 
this  report. 

Cochrane  Effective  practice  and  Organization  of  Cark  (EPOC)  Data  Bash 

The  EPOC  is  a  Cochrane  Review  Group  that  focuses  on  interventions  designed  to  improve  professional 

practice  and  the  delivery  of  effective  health  services,  such  as  various  forms  of  continuing  education,  quality 
assurance,  informatics,  and  financial,  organizational,  and  regulatory  interventions  to  help  health  care 
professionals  deliver  services  more  effectively  or  efficiently.  Specific  examples  of  relevant  interventions 
include  case  management,  revision  of  professional  roles,  use  of  multi-disciplinary  teams,  formularies,  and 
changes  in  medical  record  systems.  Interventions  aimed  at  changing  consumer  behavior  do  not  fall  within 
EPOC's  scope,  unless  both  professional  and  patient  behavior  is  affected.  The  EPOC  library  is  located  at  the 
Health  Services  Research  Unit,  University  of  Aberdeen,  Scotland. 

Like  other  Cochrane  Review  Groups,  EPOC  maintains  a  database,  or  register,  of  published  studies  that  fall 
within  its  scope.  EPOC  developed  its  register  via  a  sophisticated  search  and  review  of  relevant  Medline 
articles.  Although  the  Medline  search  is  not  yet  complete,  the  EPOC  register  remains  a  unique  source  of 
studies  on  changing  professional  practice.  At  the  time  of  this  search,  it  contained  673  records  of  studies, 
which  were  either  randomized  controlled  trials  (N=423),  controlled  clinical  trials  (N=48),  controlled  before 
and  after  studies  (N=154)  or  interrupted  time  series  (N  =  48).  A  reference  librarian  in  Aberdeen  searched  this 
register  and  the  Medline  years  not  covered  by  the  EPOC  register  for  articles  relevant  to  this  project,  using  the 
search  strategy  outlined  in  Table  2. 
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Table  2.  EPOC  Literature  Search  Strategy 


#epoc  strategy  -  subject  terms  and  methods  20/07/98 
exp  *education,continuing/ 

(educations  adj2  (program$  or  intervention?  or  meeting?  or  session?  or  strategS  or  workshop?  or  visit?)). tw. 

(behavio?r$  adj2  intervention ?).tw. 

^pamphlets/ 

(leaflet?  or  booklet?  or  poster  or  posters). tw. 
((written  or  printed  or  oral)  adj  information). tw. 
(informations  adj2  campaign). tw. 
(educations  adjl  (method?  or  material?)). tw. 
outreach. tw. 

((opinion  or  educations  or  influential)  adjl  leader?). tw. 
facilitator?. tw. 
academic  detailing. tw. 
consensus  conference?.tw. 
practice  guideline?.tw. 

(guideline?  adj2  (introducS  or  issuS  or  impact  or  effect?  or  disseminatS  or  distribut$)).tw. 

((effect?  or  impact  or  evaluatS  or  introducS  or  comparS)  adj2  training  program$).tw. 

*reminder  systems/ 

reminder?.tw. 

(recall  adj2  system$).tw. 

(prompter?  or  prompting). tw. 

algorithm?.tw. 

^feedback/  or  feedback.tw. 

(feedback  adj  1  (loop?  or  control?  or  regulaS  or  mechanism?  or  inhibS  or  system?  or  circuit?  or  sensory  or 

visual  or  audi$)).tw 

22  not  23 

chart  reviewS. tw. 

((effect?  or  impact  or  records  or  chart?)  adj2  audit). tw. 
*patient  education/ 
counsels. tw. 
compliance. tw. 
marketing. tw. 

exp  Reimbursement  mechanisms/ 
fee  for  service. tw. 
*capitation  fee/ 

*  "deductibles  and  coinsurance"/ 
cost  sharS.tw. 

(copayment?  or  co  payment?). tw. 

(prepayS  or  prepaid  or  prospective  payment?). tw. 

*hospital  charges/ 

formularS.tw. 

fundholdS.tw. 

^medicaid/ 

*medicare/ 

blue  cross. tw. 

*nurse  clinicians/ 

*nurse  midwives/ 
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Table  2.  EPOC  Literature  Search  Strategy 
(continued) 


*nurse  practitioners/ 

(nurse  adj  (rehabilitator?  or  clinician?  or  practitioner?  or  midwi$)).tw. 

*pharmacists/ 

clinical  pharmacist?. tw. 

paramedic?. tw. 

*patient  care  team/ 

(team  adj2  (care  or  treatment)). tw. 

(integrat$  adj2  (care  or  service?)). tw. 

(care  adj2  (coordinat$  or  program$  or  continuity)). tw. 

(case  adj  1  management). tw. 

exp  *ambulatory  care  facilities/ 

^ambulatory  care/ 

*home  care  services/ 

*hospices/ 

*nursing  homes/ 

*office  visits/ 

*day  care/ 

*aftercare/ 

*community  health,nursing/ 
(chang$  adjl  location?). tw. 
domiciliary. tw. 
(home  adjl  treat$).tw. 
day  surgery. tw. 
*medical  records/ 

*medical  records  systems,  computerized/ 

(information  adj2  (management  or  system?)). tw. 

*peer  review/ 

^utilization  review/ 

^physician's  practice  patterns/ 

quality  assurance. tw. 

^process  assessment/  [health  care] 

*program  evaluation/ 

*length  of  stay/ 

(early  adjl  discharg$).tw. 

offset. tw. 

triage. tw. 

near  patient  testing. tw. 
*medical  history  taking/ 
telephone/ 

(physician  patient  adj  (interaction?  or  relationship?)). tw. 
*health  maintenance  organizations/ 
managed  care.tw. 
(hospital?  adj  1  merg$).tw. 

((standard  or  usual  or  routine  or  regular  or  traditional  or  conventional  or  pattern)  adj2  care).tw. 
(program$  adj2  (reduc$  or  increas$  or  decreas$  or  chang$  or  improv$  or  modify$  or  monitor$  or  care)).tw. 
(programs  adjl  (health  or  care  or  intervention?)). tw. 

((effect?  or  impact  or  evaluatS  or  introduc$  or  compar$)  adj2  treatment  program$).tw. 
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Table  2.  EPOC  Literature  Search  Strategy 
(continued) 


((effect?  or  impact  or  evaluat$  or  introduc$  or  compar$)  adj2  care  program$).tw. 
((effect?  or  impact  or  evaluat$  or  introduc$  or  compar$)  adj2  screening  program$).tw. 
((effect?  or  impact  or  evaluat$  or  introduc$  or  compara$)  adj2  prevent$  program$).tw. 
(computer$  adj2  (dosage  or  dosing  or  diagnosis  or  therapy  or  decision?)). tw. 
((introduc$  or  impact  or  effect?  or  implements  or  computer$)  adj2  protocol?). tw. 
((effect?  or  impact  or  introduc$)  adj2  (legislation  or  regulations)). tw. 
or/ 1-2 1,24-98 

randomized  controlled  trial.pt. 
controlled  clinical  trial.pt. 
intervention  studies/ 
experiments. tw. 
(time  adj  series). tw. 

(pre  test  or  pretest  or  (posttest  or  post  test)).tw. 

random  allocation/ 

impact.tw. 

intervention?. tw. 

chang$.tw. 

evaluation  studies/ 

evaluat$.tw. 

effect?.tw. 

comparative  studies/ 

animal/ 

human/ 

114  not  115 

or/100-113 

117  not  116 

99  and  118 


The  titles  and  abstracts  of  all  studies  identified  by  this  search  were  sent  to  the  Southern  California  Evidence 
Based  Practice  Center  for  review. 
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Previous  Systematic  Reviews 

In  addition  to  the  search  described  above,  Dr.  Jeremy  Grimshaw  of  the  EPOC  identified  seven  previously 
completed  systematic  reviews  relevant  to  this  project  from  his  personal  files.  Each  review  discusses  one  or 
more  interventions  aimed  at  increasing  utilization  of  health  services.  Table  3  lists  these  seven  publications. 

Table  3.  Review  Articles 

Austin  SM,  Balas  EA.  Mitchell  JA,  Ewigman  BG.  (1994).  Effect  of  physician 

reminders  on  preventive  care:  Meta-analysis  of  randomized  clinical  trials. 
Paper  presented  at  the  Annual  Symposium  of  Computers  in  Applied  Medical 
Care:  121-124. 

Buntinx  F,  Winkens  R,  Grol  R,  Knottnerus  JA.  (1993).  Influencing  diagnostic  and 
preventive  performance  in  ambulatory  care  by  feedback  and  reminders:  A 
review.  Family  Practice,  10,  219-228. 

Grilli  R,  Freemantle  N,  Minozzi  S,  Domenighetti  G,  Finer  D.  (1998).  Impact  of  mass 
media  on  health  services  utilisation  (Cochrane  Review).  In:  The  Cochrane 
Library,  Issue  4.  Oxford:  Update  Software. 

Gyorkos  TW,  Tannenbaum  TN,  Abrahamowicz  M.  Bedard  L,  Carsley  J,  Franco  ED, 
Delage  G,  Miller  MA,  Lamping  DL,  Grover  SA.  (1994).  Evaluation  of  the 
effectiveness  of  immunization  delivery  methods.  Revue  Canadienne  de  Sante 
Publique.  85,  14-30. 

Mandelblatt  J,  Kanetsky  PA.  (1995).  Effectiveness  of  interventions  to  enhance 

physician  screening  for  breast  cancer.  The  Journal  of  Family  Practice,  40(2), 
162-171. 

Shea  S.  DuMouchel  W.  Bahamonde  L.  (1996).  A  meta-analysis  of  16  randomized 
controlled  trials  to  evaluate  computer-based  clinical  reminder  systems  for 
preventive  care  in  the  ambulatory  setting.  Journal  of  the  American  Medical 
Informatics  Association,  3(6),  399-409. 

Snell  JL,  Buck  EL.  (1996).  Increasing  cancer  screening:  A  meta-analysis.  Preventive 
Medicine,  25,  702-707. 


These  seven  reports  utilized  the  following  criteria  for  identifying  and  selecting  studies  for  review: 

Austin  and  colleagues  ( 1994)  analyzed  the  use  of  physician  reminders.  All  trials  included  in  their  meta- 
analysis were  conducted  in  a  family  or  internal  medicine  clinic  and  focused  on  cervical  cancer  screening  and 
tetanus  immunization.  The  final  review  consisted  of  studies  that  met  the  following  criteria:  a)  was  a 
randomized  controlled  clinical  trial;  b)  compared  information  or  utilization  management  intervention  in  the 
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study  group  with  no  similar  assistance  in  the  control  groups;  and  c)  evaluated  the  change  in  the  process  and  / 
or  outcome  of  patient  care. 

Buntinx  and  colleagues  ( 1993)  studied  the  effects  of  feedback  and  reminders  on  the  performance  of  doctors 
with  ambulant  patients.  The  authors  searched  the  Medline  database  electronically  for  articles  published  from 
1983  to  early  1992.  They  also  manually  searched  FAMLI  for  articles  published  from  1980  to  1990.  They 
carefully  screened  the  references  of  each  article  they  retrieved  and  obtained  additional  papers  from  personal 
collections  and  experts  in  the  field.  The  final  review  included  only  randomized  studies  that  examined  the 
effect  of  various  forms  of  feedback  on  the  performance  of  doctors  relative  to  guidelines  for  cervical  screening, 
mammogram  referral,  detection  of  occult  blood  in  feces,  and  vaccination. 

Mandelblatt  and  Kanetsky  (1995)  analyzed  interventions  to  enhance  physician  screening  for  breast  cancer. 
They  obtained  studies  from  a  Medline  search  of  articles  published  between  January  1980  and  April  1993,  and 
they  reviewed  entries  posted  to  the  Current  Contents  database  between  November  1992  and  April  1993. 
Bibliographic  references  in  retrieved  articles  were  reviewed  for  additional  citations.  The  final  publication 
included  only  concurrent  control  studies  based  in  the  United  States. 

Shea  et  al  (1996)  studied  the  use  of  computer-based  reminder  systems  for  preventive  care  in  ambulatory 
settings.  They  searched  the  Medline,  Nursing  and  Allied  Health,  and  Health  Planning  and  Administration 
databases  using  the  key  phrase  "reminder  systems."  They  subsequently  reviewed  the  reference  lists  for  the  90 
articles  retrieved  through  this  search;  this  process  identified  28  more  articles.  The  final  publication  included 
16  randomized,  controlled  studies  of  computer-based  reminder  systems  for  preventive  services  in  ambulatory 
settings. 

Snell  and  Buck  ( 1996)  reviewed  the  effects  of  interventions  directed  at  either  patients  or  physicians  on 
screening  rates  for  breast,  cervical,  and  colorectal  cancers.  They  searched  the  National  Library  of  Medicine 
database  (MeSH)  for  literature  published  from  1989  to  1994.  Six  different  searches  were  performed  using 
combinations  of  the  words  mammogram,  breast  cancer,  prevention,  health  promotion,  cancer  screening, 
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intervention,  reminder,  physician,  and  patients.  They  also  retrieved  relevant  references  listed  in 
bibliographies. 

Gyorkos  and  colleagues  ( J 994)  examined  methods  of  increasing  immunization  delivery.  They  searched  the 
Medline  and  SCISEARCH  databases  for  literature  published  from  1979  to  1992,  and  they  reviewed  the 
bibliographies  of  retrieved  articles  for  additional  references.  They  also  consulted  content  experts  to  identify 
unpublished  documentation.  The  final  publication  included  only  studies  that  met  the  following  selection 
criteria:  a)  restricted  the  target  population  to  humans  from  developed  countries;  b)  placed  no  restriction  on 
the  type  of  outcome  measured;  c)  compared  one  or  more  interventions  with  a  control  group;  and  d)  was 
published  in  French  or  English. 

Grilli  et  al  ( 1998)  analyzed  the  impact  of  mass  media  on  health  services  utilization.  They  included 
randomized  controlled  trials,  controlled  clinical  trials,  controlled  before  and  after  studies,  and  interrupted  time 
series  studies  that  met  the  EPOC  criteria  (described  earlier).  To  identify  relevant  studies,  they  searched  the 
Medline  and  Embase  databases  without  language  or  time  restriction.  The  Medline  search  was  based  upon  the 
following  terms:  health  promotion  (MeSH  descriptor);  and  communications-media  (exploded);  or  journalism 
(exploded);  or  advertising  (exploded);  or  propaganda  (exploded);  or  marketing  of  health  services  (exploded). 
The  Embase  search  was  based  upon  the  following  terms:  health  promotion  (EMTREE  descriptor);  plus 
audiovisual-equipment  (exploded);  or  mass  communication  (exploded);  or  commercial-phenomena 
(exploded);  or  publication  (EMTREE  descriptor);  or  patient  information  (EMTREE  descriptor);  or  visual 
information  (EMTREE  descriptor).  They  also  searched  the  EPOC  register,  ERIC,  and  PsychLit. 

Center  for  Quality  of  Care  Research,  Netherlands 

This  center  recently  completed  a  systematic  review  of  the  literature  studying  interventions  used  to  improve 

delivery  of  preventive  services  in  primary  care  (  Hulscher.  1999).  To  identify  relevant  articles,  they  searched 

Medline  for  literature  published  from  1968  to  1995  using  two  search  strategies.  The  first  search  used 

keywords  referring  to  implementation  combined  with  keywords  referring  to  preventive  activities  in  primary 

care  settings.  The  second  search  used  the  implementation  keywords  combined  with  keywords  referring  to  test 
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ordering  behavior,  followed  by  manual  selection  of  studies  performed  in  primary  care.  They  also  manually 
searched  the  tables  of  contents  of  21  scientific  journals,  focusing  on  studies  about  implementing  guidelines 
and  changes.  Studies  were  also  identified  through  the  EPOC  database  described  earlier. 

They  accepted  for  review  any  type  of  professional,  organizational,  financial,  or  regulatory  intervention  aimed 
at  professionals  to  improve  preventive  activities.  However,  their  search  only  included  studies  published  in 
English,  German,  or  Dutch.  The  final  publication  reviewed  58  relevant  studies,  comprising  86  comparisons 
between  intervention  and  control  groups. 

Centers  for  disease  Control  and  prevention  (CDC) 

The  CDC  recently  summarized  the  scientific  evidence  on  interventions  aimed  at  increasing  population-based 
vaccination  coverage.  (It  shared  a  draft  copy  of  the  summary  with  us.)  Interventions  included  increasing 
community  demand  for  vaccinations,  enhancing  access  to  vaccination  services,  legislating  /  regulating 
vaccinations,  and  provider-based  strategies.  To  develop  the  summary,  the  CDC  performed  electronic 
searches  of  the  Medline.  Embase,  PsychLit,  CAB  Health,  and  Sociological  Abstracts  databases.  Studies  had 
to  meet  the  following  inclusion  criteria: 

•  be  published  between  1980  and  1995 

•  address  universally  recommended  adult,  adolescent,  or  childhood  vaccinations 

•  be  a  primary  study  (rather  than  a  review) 

•  take  place  in  an  industrialized  country 

•  be  written  in  English 

•  meet  the  summary's  definition  of  interventions 

•  provide  information  on  one  or  more  outcomes  related  to  the  analytic  frameworks 

•  compare  a  group  of  persons  who  had  been  exposed  to  the  intervention  with  a  group  who  had  not  been 
exposed  or  had  been  less  exposed. 

The  CDC  searches  resulted  in  a  total  of  99  articles.  From  these  99  articles,  we  subsequently  selected  the  all 
studies  relevant  to  influenza  and  pneumococcal  vaccination  for  further  review. 
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Health  Care  Quality  improvement  Projects  (HCQIP) 

Each  U.S.  state  and  territory  is  associated  with  a  Medicare  Peer  Review  Organization  (PRO)  that  conducts 
various  research  projects.  HCFA  maintains  a  database  with  a  narrative  description  of  each  research  project, 
called  the  NPD  (Narrative  Project  Document).  An  NPD  includes  the  aims,  background,  quality  indicators, 
collaborators,  sampling  methods,  interventions,  measurement,  and  results  of  a  project. 

We  searched  the  NPD  database  using  the  following  keywords: 

•  for  influenza  vaccine  -  influenza,  flu,  immunization 

•  for  pneumococcal  vaccine  -  pneumococcal,  ppv,  immunization 

•  for  mammography  -  breast,  mammography,  preventive 

•  for  cervical  smear  cytology  -  pap,  cervical,  cervico 

•  for  colon  cancer  screening  -  colitis,  colon  cancer,  polyps,  endoscopy,  colonoscopy. 

This  search  retrieved  148  NPDs.  The  majority  addressed  influenza  vaccination;  none  covered  cervical  smear 
cytology.  The  remainder  dealt  with  pneumoccocal  vaccination,  mammography,  or  colon  cancer  screening. 

Evaluation  of  Potential  Evidence 

After  retrieving  articles  from  the  literature  sources  described  above,  we  then  reviewed  them  against  exclusion 
criteria  to  determine  whether  to  include  them  in  the  evidence  synthesis.  After  consultation  with  our  technical 
experts  (Drs.  Mittman.  Rubenstein.  and  Grimshaw),  we  created  a  one-page  screening  review  form  (Figure  2). 
The  form  consists  of  a  series  of  yes/no  questions,  formatted  into  a  checklist  (with  a  check  meaning  "yes"). 
After  evaluation  against  this  checklist,  an  article  was  either  accepted  for  further  review  or  rejected.  Two 
physician  reviewers,  each  trained  in  the  critical  analysis  of  scientific  literature,  independently  reviewed  each 
study,  abstracted  data,  and  resolved  disagreements  by  consensus.  Any  disagreements  that  remained 
unresolved  after  discussion  between  the  reviewers  were  resolved  by  a  third  party  (Dr.  Shekelle  or  Dr.  Stone). 
Project  staff  entered  data  from  the  checklists  into  an  electronic  database  used  to  track  all  studies  as  they  went 
through  the  screening  process. 
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In  order  to  be  accepted  at  this  stage,  a  study  had  to  address  one  or  more  of  the  five  services  of  interest  and  use 
one  of  the  following  study  designs:  randomized  controlled  trial,  controlled  clinical  trial,  controlled  before  and 
after  study,  or  interrupted  time  series  with  adequate  data  points.  While  we  were  primarily  searching  for  data 
relevant  to  the  Medicare  population,  we  included  studies  that  included  data  on  populations  under  age  65  at 
this  stage  to  avoid  premature  loss  of  potentially  useful  data. 

We  defined  the  study  types  according  to  the  EPOC  criteria,  described  below. 

Randomized  controlled  trial  (RCT).  A  trial  in  which  the  participants  (or  other  units)  are  definitely  assigned 
prospectively  to  one  or  two  (or  more)  alternative  forms  of  health  care,  using  a  process  of  random  allocation 
(e.g.,  random  number  generation,  coin  flips). 

Controlled  clinical  trial  (CCT).  A  trial  in  which  participants  (or  other  units)  are  either: 

a)  definitely  assigned  prospectively  to  one  or  two  (or  more)  alternative  forms  of  health  care  using  a 
quasi-random  allocation  method  (e.g.,  alternation,  date  of  birth,  patient  identifier) 

OR 

b)  possibly  assigned  prospectively  to  one  or  two  (or  more)  alternative  forms  of  health  care  using  a 
process  of  random  or  quasi-random  allocation. 

Controlled  before  and  after  study  (CBA).  A  study  in  which  the  intervention  and  control  groups  become 
involved  in  the  study  other  than  by  random  process,  and  in  which  the  baseline  period  of  assessment  is 
included  in  the  main  outcomes.  The  EPOC  uses  these  two  minimum  criteria  for  including  CBAs  in  its 
reviews: 

a)  contemporaneous  data  collection  -  data  on  the  pre-  and  post-intervention  periods  for  the  study  and 
control  sites  are  the  same 

b)  appropriate  choice  of  control  sites  -  the  study  and  control  sites  are  comparable  with  respect  to 
dominant  reimbursement  system,  level  of  care,  setting  of  care,  and  academic  status. 
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Interrupted  time  series  (ITS).  An  ITS  study  examines  data  trends  and  attributes  a  change  in  trend  to  an 
intervention.  Such  studies  can  be  either  retrospective  or  prospective.  The  EPOC  uses  these  two  minimum 
criteria  for  including  ITS  designs  in  its  reviews: 

a)  a  clearly  defined  point  in  time  at  which  the  intervention  occurred 

b)  at  least  three  data  points  before  and  three  data  points  after  the  intervention. 

Because  of  these  restrictions  on  study  design,  we  excluded  studies  that  employed  a  simple  pre/post  design. 
(A  simple  pre/post  study  design  is  one  in  which  an  intervention  is  administered  to  providers,  patients,  or 
communities,  and  the  proportion  of  persons  receiving  the  service  is  recorded  once  before  and  once  after  the 
intervention.)  Such  a  study  design  has  no  control  group;  therefore,  it  cannot  account  for  temporal  effects 
unrelated  to  the  intervention. 
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Figure  2.  Screening  Form 
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Extraction  of  Study-Level  Variables  and  Results 

After  retrieving  relevant  articles,  we  abstracted  their  data  on  a  specialized  form  (Figure  3).  The  form  contains 
questions  about  the  study  design;  the  number  and  characteristics  of  the  patients;  the  setting,  location,  and 
target  of  the  intervention;  the  intensity  of  the  intervention;  the  types  of  outcome  measures;  the  time  from 
intervention  until  outcome  measurement;  and  the  results.  We  selected  the  variables  for  abstraction  with  input 
from  the  project's  technical  experts.  Two  physicians  working  independently  extracted  data  in  duplicate,  and 
resolved  disagreements  by  consensus.  Dr.  Stone  resolved  disagreements  not  resolved  by  consensus.  As  a 
final  check  Dr.  Shekelle  reviewed  all  data  on  sample  size  and  outcomes  which  could  have  entered  into  the 
meta-regression  analysis. 

To  evaluate  the  quality  of  the  study,  we  collected  information  on  the  study  design  (with  the  hierarchy  of 
internal  validity  being  RCT,  CCT,  CBA,  and  ITS),  withdrawal/dropout  rate,  and  agreement  between  the  unit 
of  randomization  and  the  unit  of  analysis.  We  did  not  assess  studies  for  additional  criteria  with  empirical 
support  for  their  association  with  bias  (blinding  and  concealment  of  allocation)  (Moher,  1998)  because  those 
were  not  feasible  in  many  studies  of  these  types  of  interventions.  The  primary  outcome  consisted  of  the 
proportion  of  patients  who  received  the  service  before  and  after  the  intervention. 
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Figure  3.  Abstraction  Form 
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2.  First  Author: 
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Figure  3. 


Abstraction  Form  (continued) 


8.    What  best  characterizes  the 
setting  of  the  intervention? 
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9.  What  best  characterizes  the 
geographic  setting  of  the 
intervention? 

Mainly  rural  1 
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Not  sure  4 

10.  In  what  health-care  practice 
settings  did  the  intervention 
occur? 
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Outpatient  2 

Both  hospital  and  outpatient  3 

Nursing  home   4 

Not  applicable  9 

1 1.  What  best  describes  the 
reimbursement  system 
of  the  care  in  which 

the  intervention  occurred? 

Fee-for-service  1 

HMO  2 

Managed  care,  not  HMO  3 

Mixed  reimbursement 

systems  4 

Other  5 

Not  applicable  9 
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Figure  3.  Abstraction  Form  (continued) 

13.  What  best  characterizes  the  intervention?  Check  all  that  apply. 
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Figure  3.  Abstraction  Form  (continued) 
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Figure  3.  Abstraction  Form  (continued) 


Group  3 
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Figure  3.  Abstraction  Form  (continued) 


16.  How  many  analyzed  units  were  enrolled  and  followed  up  in. 

Group  1  Entered   


Followed 
Group  2  Entered 

Followed 
Group  3  Entered 

Followed 
Group  4  Entered 

Followed 


17.  What  was  the  unit  of  allocation? 

Patient.-.  1 

Provider  2 

Organization  3 

Community  or 

geographic  area  4 

Not  applicable  9 

18.  What  was  the  unit  of  analysis? 

Patient  1 

Provider  2 

Organization  3 

Community  or 

geographic  area  4 

Not  applicable  9 

19.  If  the  unit  of  allocation  and  the  unit  of  analysis 
are  not  the  same,  was  any  statistical  correction 
made  for  clustering? 

Yes  2 

No  2 

Not  applicable  9 

20.  Was  there  a  sample-size  justification  or  power  calculation? 

Yes   1 

No  2 

21.  What  outcomes  were  measured? 

Proportions  1 

Other  2 
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Figure  3.  Abstraction  Form  (continued) 

22.  When  were  the  outcomes  measured  relative  to  after  the  start  of  the  intervention? 
 weeks 


23.  Costs  analyzed?   Y/N  (If  Yes,  give  to  Erin) 

24.  Describe  the  outcomes:  (If  not  a  proportion,  give  to  Erin) 


Group  1 


Group  2 


Group  3 


Group  4 


Group  1 


Group  2 


Group  3 


Mammogram 


Proportion  (%) 
before  intervention 


Proportion  {%) 
after  intervention 


p-value 


Proportion  (%) 
before  intervention 


Cervical  smear  cytology 

Proportion  (%)  p-value 
after  intervention 


Comparison 
group 


Comparison 
group 


Group  4 


45 


Figure  3.  Abstraction  Form  (continued) 


Group  1 


Group  2 


Group  3 


Group  4 


Group  1 
Group  2 
Group  3 
Group  4 


Colon  Cancer  Screen  (FOBT) 

Proportion  (%)         Proportion  (%)  p-value  Comparison 

before  intervention     after  intervention  group 


Colon  Cancer  Screen  (colon  visualization) 

Proportion  (%)         Proportion  (%)  p-value  Comparison 

before  intervention     after  intervention  group 


46 


Figure  3.  Abstraction  Form  (continued) 


Group  1 


Group  2 


Group  3 


Group  4 


Group  1 


Group  2 


Group  3 


Group  4 


Flu  Shot 

Proportion  (%)         Proportion  (%)  p-value  Comparison 

before  intervention     after  intervention  group 


Pneumococcal  Vaccine 

Proportion  (%)         Proportion  (%)  p-value  Comparison 

before  intervention     after  intervention  group 
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Figure  3.  Abstraction  Form  (continued) 
Additional  instructions 

Intensity:  Length  of  time  in  minutes  for  each  unit  of  intervention,  e.g.  60  minute  educational  session,  1 
minute  TV  spot,  5  minute  educational  visit  with  detailing. 

Duration:  Length  of  time  in  days  from  start  of  intervention  to  end  of  intervention,  e.g.  TV  spots  ran  for  15 
days,  educational  session  occurred  only  once  (1  day),  reminders  started  June  1  and  ended  June  30  (30  days). 

Number  of  units  of  intervention:  Number  of  times  the  intervention  occurred  for  each  target,  e.g.  1 
educational  session  each  week  for  5  weeks  for  each  provider  (5  units),  2  reminders  sent  to  each  patient  (2 
units),  1  brochure  given  to  each  patient  (1  unit). 

Medium/Delivery  vehicle  of  intervention.  Write  down  number(s)  from  list  below  (3  numbers  max): 

1 .  In  person 

2.  By  telephone 

3.  In  group 

4.  Radio 

5.  TV 

6.  —  : 

7.  Electronic 

8.  — 

9.  — 

10.  Poster 

11.  Mail 

12.  Other 

13.  Printed  material  (e.g.  newsprint,  brochure,  computer  printout) 

Content:  Was  there  mention  that  the  content  was  tailored  to  the  audience  (e.g.  large  print,  6lh  grade  level 
writing,  etc)?  Write  Y  for  Yes  and  N  for  No. 
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Expert  Panel  Review  of  Evidence  Report 

We  presented  the  draft  evidence  report  to  a  panel  of  experts  (Table  4)  for  feedback  and  discussion  on  March 
17,  1999  at  a  meeting  organized  by  the  Partnership  for  Prevention.  During  this  meeting  we  reviewed  our 
methods  and  preliminary  results. 

In  the  draft  report,  we  had  restricted  our  sample  to  studies  that  presented  data  on  individuals  age  65  or  over. 
Because  of  this  restriction,  we  only  had  enough  studies  to  perform  meta-regression  analyses  for  influenza 
vaccination.  The  expert  panelists  recommended  that  we  increase  our  sample  size  so  that  we  could  perform 
additional  quantitative  analyses.  They  suggested  that  we  do  so  by  going  back  to  the  literature  and  including 
all  studies  of  adult  populations  (i.e.,  studies  that  examined  patients  under  age  65),  under  the  assumption  that 
the  age  of  the  study  population  would  have  minimal  or  no  impact  on  the  effectiveness  of  interventions.  They 
also  suggested  that  we  contact  managed  care  organizations  to  determine  whether  there  was  additional, 
unpublished  information  available  that  met  our  eligibility  criteria. 

In  addition,  the  expert  panelists  recommended  that  we  combine  the  interventions  for  pneumonia  vaccination 
and  influenza  vaccination  into  a  single  analysis  because  such  interventions  were  unlikely  to  differ  in  their 
clinical  effectiveness.  They  also  suggested  that  we  perform  analyses  that  would  examine  the  interventions  as 
a  function  of  the  year  the  intervention  was  attempted,  the  baseline  rate  of  the  procedure  prior  to  the 
intervention,  and  the  intensity  of  the  intervention. 
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Table  4.  Expert  Panel 


John  Burton,  MD 
Director,  Division  of  Geriatric 
Medicine  and  Gerontology 
Johns  Hopkins  Geriatric  Center 
Johns  Hopkins  University 

Ned  Calonge,  MD,  MPH 

Director,  Geriatric  Medicine  and  Gerentology 

Johns  Hopkins  Bayview  Medical  Center 

Gary  Dennis,  MD,  FAACS 

President,  National  Medical  Association 

Marge  Drugay,  ND,  RN,  C 
Drugay  and  Associates 

Jonathan  Fielding,  MD,  MPH,  MBA 

Director  of  Public  Health 

LA  County  Dept.  of  Health  Services 


Russell  Morgan,  DrPH 
President,  SPRY  Foundation 

Don  Nielsen,  MD 

Senior  VP  for  Quality  Leadership 

American  Hospital  Association 

Michael  Rosnick,  MD 

Medical  Director,  HMA  Health  Plan 

Randolph  D.  Smoak,  Jr.,  MD 
Chair,  AM  A  Board  of  Trustees 
American  Medical  Association 

Kurt  C.  Stange,  MD,  PhD 

Associate  Professor  of  Family  Medicine 

Cancer  Center  Director  for  Prevention  and  Control 

Case  Western  Reserve  University 


Lisa  Foley,  JD 

Senior  Analyst,  Public  Policy  Institute 
American  Association  of  Retired  Persons 


Steven  M.  Teutsch,  MD,  MPH 
Senior  Research  Scientist 
Merck  &  Co,  Inc. 


William  E.  Golden,  MD 
President,  AHQA 

University  of  Arkansas  for  Medical  Sciences 

Jessie  C.  Gruman.  PhD 
Executive  Director 

Center  for  the  Advancement  of  Health 

William  Kavesh,  MD,  MPH 
VA  Hospital,  Geriatric  Clinic 


Jan  Towers,  PhD,  NP-C,  CRNP 
Director  of  Government  Affairs,  Practice 
and  Research 

American  Academy  of  Nurse  Practitioners 

Steven  H.  Woolf,  MD,  MPH 
Professor  of  Family  Medicine 
Medical  College  of  Virginia 
Virginia  Commonwealth  University 


William  L.  Roper,  MD,  MPH 
Chairman,  Partnership  for  Prevention 
Dean,  School  of  Public  Health 
University  of  North  Carolina  @  Chapel  Hill 
(Expert  Panel  Chair) 


Disclaimer:  Participation  as  an  Expert  Panelist  does  not  indicate  consensus  with  the 
recommendations  of  this  evidence  report. 
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Statistical  Methods 

Prior  to  our  analysis,  we  entered  all  data  on  outcomes  and  interventions  into  the  statistical  package  SAS.  In 
the  analysis  itself,  we  sought  to  answer  the  following  questions: 

1 .  What  is  the  absolute  effectiveness  of  each  intervention  in  improving  the  use  of  clinical  preventive  and 
screening  services? 

2.  What  is  the  relative  effectiveness  of  each  intervention  in  improving  the  use  of  clinical  preventive  and 
screening  services? 

3.  How  do  important  covariates,  such  as  the  target  population  (e.g.  low  income  or  other  vulnerable 
populations)  or  the  setting  in  which  the  intervention  is  applied  (e.g..  academic  versus  non-academic 
practice  and  managed  care  versus  fee-for-service  systems),  impact  the  effectiveness  of  the  interventions? 

4.  What  is  the  relative  cost  effectiveness  of  each  intervention  in  increasing  the  use  of  clinical  preventive  and 
screening  services? 

5.  What  elements  appear  to  be  instrumental  in  determining  the  success  of  each  intervention? 
META-REGRESSION  ANALYSIS 

We  first  retrieved  all  studies  that  assessed  the  effects  of  an  intervention  or  interventions  relative  to  either  a 
group  that  received  usual  care  or  to  a  control  group.  We  then  fit  a  series  of  meta-regressions  to  these  studies 
(Struck.  1993).  The  basic  data  matrix  for  the  meta-regressions  was  as  follows.  Each  study  with  a  single 
intervention  arm  contributed  four  observations  corresponding  to  the  cells  of  a  two-by-two  table  of  treatment 
by  outcome  (control  and  intervention  cases  that  received  the  preventive  or  screening  service;  control  and 
intervention  cases  that  did  not)  to  a  weighted  logistic  regression  that  predicted  receipt  of  the  preventive  or 
screening  service  or  not.  An  observation's  weight  was  equal  to  the  number  of  individuals  belonging  to  the 
corresponding  cell.  Studies  which  had  more  than  one  intervention  contributed  an  additional  pair  of 
observations  (those  who  did  not  and  those  who  did  receive  the  service  in  the  intervention  group  respectively) 
for  each  additional  intervention.  For  example,  a  study  that  had  three  intervention  arms  contributed  eight 
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observations  to  the  meta-regression,  two  for  the  control  group,  two  for  the  first  intervention,  two  for  the 
second  intervention,  and  two  for  the  third  intervention. 

To  assess  the  statistical  significance  of  each  type  of  intervention,  or  of  the  interaction  between  treatment  and  a 
particular  covariate  of  interest,  for  example  whether  intervening  worked  better  for  particular  subpopulations, 
we  constructed  specific  models  that  contained  both  an  intervention  component  indicator  or  specific  covariate 
by  treatment  interaction  indicator,  and  indicator  variables  for  each  study.  The  inclusion  of  study  indicators 
controlled  for  all  measured  study  characteristics  and  all  unmeasured  ones,  and  is  akin  to  fitting  a  fixed  effects 
model.  Each  model  produced  odds  ratios  for  covariates  of  interest  that  are  adjusted  for  all  measured  and 
unmeasured  study  level  differences.  We  present  both  adjusted  odds  ratios  versus  control  or  usual  care,  and 
versus  the  "average"  intervention  across  all  studies  as  explained  below. 

Question  l :  Absolute  effectiveness  of  each  intervention 

To  assess  the  absolute  effectiveness  of  each  intervention,  we  categorized  each  intervention  as  to  which 
intervention  components  it  contained.  For  example,  one  intervention  might  contain  both  patient  reminders 
and  provider  reminders  while  another  had  patient  financial  incentives  and  organizational  change.  An  overall 
treatment  indicator  and  individual  intervention  component  indicators  were  constructed.  For  example,  if  an 
intervention  contained  both  patient  and  provider  reminders,  the  associated  dummy  variables  for  these 
components  were  set  equal  to  one  for  the  pair  of  observations  corresponding  to  that  intervention  arm  and  were 
zero  otherwise.  In  addition,  the  overall  treatment  indicator  was  set  equal  to  one  for  all  intervention 
observations.  Thus  the  data  matrix  consisted  of  an  overall  treatment  indicator,  indicator  variables  for  all 
intervention  components  that  had  sufficient  number  of  observations  to  be  included  in  the  model,  and  study 
level  indicators  as  discussed  above  to  control  for  all  measured  and  unmeasured  study  level  differences. 

The  outcome  of  this  model  is  twofold.  First,  odds  ratios  for  the  effectiveness  of  each  intervention  component 
at  improving  the  use  of  clinical  and  preventive  services  as  compared  to  the  control  or  usual  care  situation  and 
adjusted  for  all  other  effects  are  produced.  For  example,  we  estimate  the  odds  ratio  of  the  effect  of  patient 
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reminders  as  compared  to  the  control  or  usual  care  situation  on  the  use  of  services,  and  provide  a  95% 
confidence  interval  for  that  estimate.  Second,  the  marginal  effect  of  each  intervention  component  as 
compared  with  the  "average"  intervention  across  all  interventions  in  our  study  that  do  not  contain  that 
particular  component  can  be  estimated.  These  latter  results  are  relevant  for  our  second  research  question  and 
will  be  discussed  in  its  context  below.  The  two  sets  of  odds  ratios  are  also  related  mathematically  as  will  be 
explained  below. 

We  ran  separate  models  for  each  of  the  following  preventive  and  screening  services: 

•  immunization  (this  model  combined  studies  for  both  influenza  vaccination  and  pneumococcal 
vaccination) 

•  screening  mammography 

•  cervical  smear  cytology  screening. 

•  fecal  occult  blood  testing  (for  colon  cancer  screening) 

We  were  unable  to  perform  a  meta-regression  analysis  on  the  use  of  colon  visualization  (i.e.,  sigmoidoscopy, 
colonoscopy,  or  barium  enema)  due  to  the  insufficient  number  of  studies  on  this  topic. 

Question  2:  Relative  effectiveness  of  each  intervention 

We  identified  two  possible  sources  of  information  for  determining  the  relative  effectiveness  of  the 
interventions:  direct  evidence,  which  consisted  of  eligible  studies  that  directly  compared  interventions  to  one 
another,  and  indirect  evidence,  which  consisted  of  results  from  the  meta-regression  models  described  above. 
The  direct  evidence  was  limited  because  few  studies  met  our  criteria,  and  those  that  did  usually  performed  the 
comparisons  of  interest  only  once.  Hence,  there  was  generally  only  a  single  data  point  available. 
Furthermore,  the  paucity  of  such  studies  made  interpretation  of  the  effects  of  the  intervention  difficult  due  to 
the  presence  of  other  covariates.  For  example,  the  sole  study  of  a  particular  intervention  versus  another  may 
have  taken  place  only  in  an  HMO  setting  among  non-elderly  adults.  To  infer  that  this  relationship  between 
the  interventions  would  be  similar  in  a  fee-for-service  situation  is  not  supportable. 
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Our  analysis  of  the  indirect  evidence  for  the  relative  effectiveness  of  the  interventions  consisted  of  assessing 
the  results  of  the  meta-regressions  discussed  above  in  two  ways.  First,  we  compared  the  adjusted  odds  ratio 
of  effectiveness  for  individual  interventions  versus  control.  Second,  we  calculated  the  adjusted  odds  ratio  for 
the  marginal  effect  of  an  intervention  component  versus  the  "average"  intervention  across  all  studies  that  did 
not  contain  that  intervention  component. 

To  clarify  the  analysis  discussion,  we  use  the  mammogram  intervention  meta-regression  model  as  an 
example.  A  simple  meta-regression  containing  just  a  treatment  indicator  and  study  level  indicator  variables 
produces  an  estimated  treatment  odds  ratio  as  compared  to  the  control  or  usual  care  group  of  1.67  with  a  95% 
confidence  interval  of  (1.57,  1.77).  The  intervention  model  contains  the  overall  treatment  indicator  and  as 
many  of  the  seven  intervention  component  indicators  for  provider  education,  patient  education,  feedback, 
patient  financial  incentives,  provider  reminders,  patient  reminders,  and  organizational  change  as  sample  size 
allowed.  In  effect  this  model  is  parceling  out  the  1.67  odds  ratio  across  these  intervention  components.  The 
odds  ratio  for  patient  reminders  versus  the  control  or  usual  care  setting  is  2.57,  and  is  3.57  for  patient  financial 
incentives.  We  note  in  comparison  that  the  intervention  with  the  lowest  odds  ratio  is  patient  education  at 
1.31,  and  see  that  the  overall  treatment  odds  of  1.67  is  in  the  middle  of  the  distribution  of  individual 
intervention  component  effects.  Interpreting  these  odds,  we  can  conclude  that  an  intervention  group  with 
patient  reminders  has  2.57  times  the  odds  of  a  group  subject  to  no  intervention  or  usual  care  of  receiving  the 
service.  In  contrast,  patient  financial  incentives  will  increase  the  odds  of  receiving  the  service  by  a  factor  of 
3.57.  The  marginal  odds  ratio  of  patient  reminders  versus  the  "average"  intervention  without  patient 
reminders  across  all  the  studies  is  1.31,  and  the  marginal  odds  ratio  for  patient  financial  incentives  is  1.81.  So 
adding  either  of  these  intervention  components  to  an  "average"  intervention  that  does  not  contain  the 
particular  component  increases  the  odds  of  receiving  the  service.  We  note  that  the  ratio  of  the  odds  versus 
control  (2.57/3.57=0.72)  equals  the  ratio  of  the  marginal  odds  ratios  (1.31/1.81=0.72)  as  expected. 

In  addition  to  considering  the  marginal  effectiveness  of  each  intervention  component,  we  determined  the 

treatment  effect  of  an  intervention  consisting  of  a  single  component  versus  an  intervention  consisting  of  many 
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components.  To  do  so,  in  each  service  setting,  we  fit  a  simple  meta-regression  that  contained  an  overall 
treatment  indicator  equaling  one  for  all  intervention  observations  and  zero  otherwise,  a  multiple  intervention 
treatment  indicator  equaling  one  for  all  intervention  observations  that  had  multiple  components  and  zero 
otherwise,  and  study  level  indicators  to  control  for  all  measured  and  unmeasured  study  level  differences.  For 
these  meta-regressions.  we  present  both  the  odds  ratios  for  single  and  multiple  component  interventions 
versus  the  control  or  usual  care  setting,  and  the  marginal  odds  ratio  of  adding  additional  components  to  an 
"average"  intervention  with  a  single  component. 

Question  3:  Important  cqvariates 

To  assess  the  impact  of  important  covariates  on  the  effectiveness  of  the  interventions,  we  again  turned  to  two 
sources  of  information:  direct  evidence  and  indirect  evidence.  We  defined  direct  evidence  as  all  studies  that 
assessed  the  effectiveness  of  an  intervention,  stratified  by  the  covariate  of  interest.  (For  example,  the 
effectiveness  of  an  intervention  in  an  HMO  setting  versus  a  fee-for-service  setting,  or  the  effectiveness  of  an 
intervention  among  a  low  income  population  versus  a  non-low  income  population)  However,  we  were  unable 
to  identify  any  such  studies  given  the  level  of  detail  in  our  data  collection. 

We  initially  defined  indirect  evidence  to  be  the  results  of  meta-regression  analysis  that  examined  the 
covariates  of  interest  in  a  covariate  by  intervention  interaction.  For  example,  we  hoped  to  answer  how  well 
patient  reminders  worked  for  low  income  patients  as  compared  to  how  well  patient  reminders  worked  for  non- 
low  income  patients.  However,  there  were  too  few  studies  to  allow  us  to  enter  covariates  into  our  model  in 
this  fashion.  Therefore,  we  were  limited  to  assessing  the  effect  of  the  covariate  by  treatment  across  all 
interventions  within  a  particular  service  (such  as  screening  mammography  or  immunizations).  Even  with  this 
modification,  we  found  there  were  too  few  data  to  develop  a  model  for  most  covariates  and  services. 
However,  for  each  service  and  for  those  covariates  for  which  adequate  sample  size  existed,  we  fit  a  series  of 
simple  meta-regressions  with  just  an  overall  treatment  indicator  and  a  covariate  by  treatment  interaction 
dummy.  These  models  are  similar  to  those  fit  to  determine  the  effectiveness  of  multiple  component 
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intervention  as  opposed  to  single  component  intervention  discussed  in  the  context  of  the  second  research 
question  above.  For  example,  in  the  mammogram  studies,  we  fit  a  meta-regression  with  an  overall  treatment 
indicator  and  an  indicator  variable  that  equaled  one  for  all  intervention  observations  that  were  part  of  studies 
of  low  income  patients.  The  results  indicate  how  well  the  treatment  works  in  studies  that  focused  on  low 
income  settings  as  compared  to  ones  that  did  not. 

Question  4:  Cost  effectiveness 

To  assess  the  cost  effectiveness  of  the  interventions,  we  first  determined  whether  each  study  presented  cost 
data.  Ideally,  we  would  be  able  to  derive  direct  evidence  of  cost  effectiveness  from  studies  that  compared 
interventions  head-to-head  for  both  effectiveness  and  cost.  As  we  noted  above,  however,  few  studies 
performed  more  than  one  comparison  of  effectiveness,  which  made  drawing  generalizations  problematic.  The 
number  of  such  studies  that  reported  costs  was  even  fewer.  Therefore,  we  were  not  able  to  perform 
quantitative  analyses  on  cost  effectiveness  and  chose  to  summarize  these  studies  qualitatively  instead. 

Question  5:  Elements  instrumental  to  the  success  of  each  intervention 

To  identify  which  elements  appeared  to  be  instrumental  in  determining  the  success  of  the  interventions,  we 
assessed  each  study  for  the  factors  postulated  to  increase  the  effectiveness  of  any  intervention,  as  shown  in 
our  conceptual  model  (Figure  1 ).  These  factors  included  use  of  social  influence/  opinion  leader  involvement; 
marketing/outreach;  high  visual  appeal/  clarity;  collaboration/  teamwork;  design  based  on  needs,  barriers, 
incentives,  assessments  or  theory:  top  management  support;  and  active  learning  strategies.  We  extracted  data 
on  the  intensity  of  the  intervention  and  the  underlying  screening  or  preventive  service  rate  in  the  population. 
This  underlying  rate  is  often  estimated  by  the  control  or  usual  source  of  care  group  rate.  To  assess  the  effects 
of  these  factors,  our  goal  was  to  conduct  a  meta-regression  analysis  similar  to  that  outlined  above  for  the  first 
three  research  questions. 

Regarding  the  intervention  factors  postulated  to  increase  the  effectiveness  of  any  intervention,  we  fit  meta- 
regression  models  for  each  screening  setting  that  were  similar  in  form  to  the  intervention  component 
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regressions  discussed  above  with  respect  to  the  first  research  question.  Our  models  contained  an  overall 
treatment  indicator,  indicator  variables  for  each  intervention  factor  which  equaled  one  if  the  intervention 
contained  that  factor  and  zero  otherwise,  and  study  level  indicators  to  control  for  measured  and  unmeasured 
differences  across  the  studies.  The  results  of  these  models,  specifically  the  odds  ratios  versus  control  and  the 
marginal  odds  ratios,  are  interpreted  in  the  same  manner  as  discussed  above. 

Unfortunately,  data  limitations  curtailed  our  work  in  the  intensity  and  control  rate  investigations.  We  fit  some 
preliminary  models  in  both  settings  that  were  similar  to  the  covariate  by  treatment  models  discussed  in  the 
context  of  the  third  research  question  above.  For  example,  in  the  mammogram  setting,  we  fit  a  regression 
with  an  overall  treatment  indicator  and  then  treatment  by  control  rate  categories  (less  than  20%;  20-40%;  40- 
60%;  greater  than  60%),  with  these  categories  chosen  based  on  both  clinical  expertise  and  the  empirical 
distribution  of  rates  observed  across  studies.  Unfortunately,  a  large  number  of  studies  were  missing  data,  and 
necessitated  that  a  missing  category  be  included  in  the  model.  We  conducted  a  sensitivity  analysis  of  various 
methods  to  impute  the  missing  data.  However,  the  results  from  this  analysis  were  counter-intuitive.  We  had 
expected  that  the  treatment  effect  would  decrease  as  the  underlying  population  use  of  services  increased,  as 
the  intervention  could  not  improve  an  already  high  service  rate  much.  We  did  not  observe  this  to  be  the  case, 
nor  were  the  results  in  any  way  consistent  across  the  different  service  settings.  Analogous  problems  were 
encountered  in  the  intensity  analyses.  The  large  amount  of  unreported  data,  and  the  difficulty  of  transforming 
different  intensity  metrics  to  a  common  scale  were  insurmountable. 
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RESULTS 


Identification  of  Evidence 

Figure  4  describes  the  flow  of  evidence  from  the  original  sources  to  final  acceptance.  The  EPOC  literature 
searches  identified  264  potentially  relevant  articles,  the  CDC  summary  contained  99  articles  on 
immunizations,  and  the  review  by  Hulscher  and  colleagues  from  the  Center  for  Quality  of  Care  Research 
contained  58  articles.  In  addition,  the  systematic  reviews  identified  by  Dr.  Grimshaw  contained  a  total  of  86 
references,  and  the  database  for  the  Health  Care  Quality  Improvement  Projects  yielded  149  potentially 
relevant  Narrative  Project  Descriptions  (NPDs).  After  eliminating  107  duplicate  articles,  549  potentially 
relevant  sources  of  information  remained,  which  subsequently  underwent  screening.  A  Venn  diagram 
describing  the  original  sources  of  each  of  these  articles  is  presented  in  Figure  5.  (The  "review"  articles  on  this 
diagram  refer  to  the  seven  articles  that  Dr.  Grimshaw  supplied  to  the  project.)  A  total  of  187  articles,  all 
published  between  1979  and  1999,  met  the  screening  criteria;  these  are  included  in  the  remaining  descriptions 
and  analyses. 

Distribution  of  Evidence 

Table  5  presents  the  187  studies  stratified  by  service  and  broad  characterization  of  intervention.  Note  that 
some  studies  addressed  more  than  one  service  or  combined  several  interventions;  therefore,  the  total  sums  to 
more  than  187. 

The  intervention  with  the  greatest  number  of  studies  was  patient  reminders  (131  interventions),  followed 
closely  by  patient  education  (122  interventions).  Eighty-two  studies  employed  provider  reminders,  while  69 
studies  used  provider  education.  Once  again,  these  categories  were  not  mutually  exclusive.  For  example, 
physicians  could  receive  both  education  and  reminders  simultaneously  in  a  study.  Organizational 
change/quality  improvement,  financial/administrative  incentives,  and  media  campaigns  were  studied  less 
frequently  than  the  other  interventions. 
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As  is  clear  from  the  above  numbers,  patients  were  most  often  the  targets  of  interventions:  They  were  targeted 
in  a  total  of  179  studies,  while  1 13  studies  intervened  at  the  provider  level.  In  almost  all  of  the  latter,  the 
provider  was  a  nurse  or  physician.  Thirty-three  studies  targeted  the  community,  either  through  mass  mailings 
or  media  campaigns. 

The  most-studied  procedure  was  influenza  vaccine  with  78  interventions  reported.  Mammography  was  also 
studied  frequently,  with  76  studies  passing  our  review  guidelines.  In  addition,  65  studies  on  cervical  smear 
cytology,  41  studies  on  colon  cancer  screening  using  the  Fecal  Occult  Blood  Test  (FOBT),  and  18  studies  on 
pneumococcal  vaccine  met  our  screening  criteria  and  are  included  in  this  report.  Only  ten  studies  that  met  our 
criteria  addressed  colon  visualization. 
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Table  5.  Interventions  by  Service 


Total  number  of  articles 

187 

Intervention 

Service 

#  of  studies* 

Educational 

Influenza 

55 

Pneumonia 

10 

Mammogram 

70 

Cervical  smear  cytology 

32 

Colon  cancer 

35 

Feedback 

Influenza 

5 

Pneumonia 

2 

Mammogram 

9 

Cervical  smear  cytology 

7 

Colon  Cancer 

6 

Financial/administrative  intervention  or  incentives 

Influenza 

18 

Pneumonia 

2 

Mammogram 

20 

Cervical  smear  cytology 

5 

Colon  Cancer 

7 

Reminders 

Influenza 

62 

Pneumonia 

13 

Mammogram 

74 

Cervical  smear  cytology 

38 

Colon  cancer 

45 

Organizational  Change,  quality  improvement, 
redesign/reengineering 

Influenza 

29 

Pneumonia 

7 

Mammogram 

33 

Cervical  smear  cytology 

12 

Colon  Cancer 

12 

Mass  media,  other  community-level  interventions 

Influenza 

22 

Pneumonia 

1 

Mammogram 

16 

Cervical  smear  cytology 

2 

Regulatory 

Influenza 

1 

*  The  number  of  studies  in  this  column  will  not  sum  up  to  the  total  number  of  articles,  as  some  articles 
appear  more  than  once. 
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Description  of  Evidence 

The  evidence  tables  in  Appendix  1  present  descriptive  information  for  each  study  meeting  our  acceptance 
criteria: 

•  the  author,  year,  country  of  origin,  and  study  design; 

•  the  age  and  vulnerable  population  targeted  in  the  study; 

•  the  target  of  the  intervention  (patients,  provider  type,  organizations,  communities); 

•  the  study's  setting  (academic  or  nonacademic),  the  geographic  setting  (urban/suburban  or  rural),  and 
the  setting's  reimbursement  system  (HMO,  fee-for-service,  mixed); 

•  the  two  interventions  being  compared  (e.g..  control  versus  patient  education,  provider  reminder 
versus  provider  reminder  plus  patient  education); 

•  the  characteristics  of  the  two  interventions  (population  size  [N],  baseline  rate,  and  follow-up  rate); 
and 

•  the  absolute  rate  increase  when  comparing  the  two  interventions. 

A  separate  evidence  table  is  presented  for  each  service:  influenza  vaccination,  pneumococcal  vaccination, 
mammography  screening,  cervical  smear  cytology,  colon  cancer  screening  (visualization),  and  colon  cancer 
screening  (fecal  occult  blood  test). 

Quality  of  Evidence 

Of  the  218  separate  studies  included  in  our  analysis,  136  were  randomized  clinical  trials,  24  were  controlled 
clinical  trials,  58  were  controlled  before/after  studies,  and  none  were  interrupted  time  series.  Thus,  over  one- 
third  of  the  studies  did  not  use  the  study  design  (RCT)  with  the  strongest  internal  validity.  Only  eight  studies 
reported  results  at  the  provider  level;  the  rest  reported  results  at  the  patient  level.  However,  more  than  half  of 
the  studies  used  either  the  provider,  the  organization,  or  the  community  as  the  unit  of  allocation.  Because 
none  of  these  studies  corrected  for  the  potential  clustering  of  patients  within  one  of  these  larger  units,  they 
tended  to  underestimate  the  variance  in  the  estimate  of  the  effect  of  the  intervention.  In  addition,  many 
studies  lacked  the  basic  information  necessary  for  inferential  statistical  analysis.  This  was  particularly  true 
for  the  NPDs.  which  generally  did  not  report  any  statistical  analyses  testing  for  differences  among  groups. 
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Hence,  our  ability  to  make  inferences  about  differences  among  interventions,  both  for  individual  studies  and 
for  pooled  analyses,  was  limited. 

Description  of  Results 

Question  1 :  Absolute  effectiveness  of  each  intervention 

What  is  the  absolute  effectiveness  of  each  intervention  in  improving  the  use  of  clinical  preventive  and 
screening  services? 

We  identified  the  following  studies  as  being  relevant  to  Question  1 : 

•  78  controlled  comparisons  of  interventions  to  increase  the  use  of  influenza  immunization 

•  18  controlled  comparisons  to  increase  the  use  of  pneumococcal  immunization 

•  76  controlled  comparisons  to  increase  the  use  of  mammography 

•  65  controlled  comparisons  to  increase  the  use  of  cervical  smear  cytology 

•  41  controlled  comparisons  to  increase  fecal  occult  blood  testing  for  colon  cancer 

•  10  controlled  comparisons  to  increase  the  use  of  methods  for  colon  visualization. 

META-REGRESSION 

Table  6:  Effectiveness  of  Interventions  to  Improve  the  Use  of  Clinical  and  Preventive  Services  presents  the 
results  of  the  meta-regression  analysis  for  immunizations,  mammography,  cervical  smear  cytology,  and  colon 
cancer  screening.  The  table  reports  the  odds  ratio  for  improvement  in  the  receipt  for  each  type  of  intervention 
versus  the  control  or  usual  care  situation,  adjusted  for  other  interventions  and  study  level  differences,  and 
ordered  within  service  from  the  most  effective  intervention  to  the  least.  It  also  presents  the  95%  confidence 
interval.  All  interventions  for  mammography  had  statistically-significant  adjusted  odds  ratios.  All 
interventions  for  cervical  smear  cytology,  colon  cancer  screening,  and  immunization  also  had  statistically 
significant  adjusted  odds  ratios,  with  the  exception  of  feedback.  Therefore  we  can  conclude  that,  with  the 
exception  of  feedback,  all  interventions  that  we  had  adequate  sample  size  to  include  in  the  model  are  effective 
in  improving  the  use  of  clinical  preventive  and  screening  services  when  compared  to  the  control  situation. 
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We  identified  some  consistent  patterns  across  all  four  regressions.  First,  organizational  change  was 
consistently  one  of  the  most  or  the  most  effective  interventions  at  increasing  use  of  the  clinical  and  preventive 
services.  Second,  patient  financial  incentives  were  also  highly  effective.  Third,  patient  reminders 
demonstrated  a  relatively  consistent  effect  across  all  services,  as  did  patient  education.  There  was  also  some 
evidence  that  personalized  reminders  (or  ones  signed  by  the  patient's  physician)  are  more  effective  than 
generic  reminders  (Larson,  1982;  Hardcast,  1983;  Hoggs,  1998).  Finally,  feedback  appeared  to  be  a  relatively 
ineffective  intervention,  as  it  was  statistically  beneficial  only  for  increasing  screening  mammography. 

Odds  ratios  should  not  be  confused  with  risk  ratios  and,  as  pre-intervention  utilization  rates  rise,  odds  ratios 
begin  to  overestimate  the  increase  in  expected  improvement  due  to  the  intervention.  Therefore,  in  Table  7  we 
provide  the  general  expected  improvement  in  outcomes  as  it  varies  both  by  the  pre-intervention  rate  and  the 
odds  ratio.  For  example,  if  the  pre-intervention  rate  of  use  of  mammography  was  40%,  then  implementing  an 
intervention  with  an  adjusted  odds  ratio  of  2.6  (such  as  patient  reminders)  would  be  expected  to  yield  an 
average  improvement  in  the  post-intervention  rate  to  63%  (risk  ratio=1.6).  Similarly,  if  the  pre-intervention 
rate  of  influenza  immunization  was  60%,  then  implementing  an  intervention  with  an  adjusted  odds  ratio  of  4.3 
(such  as  provider  reminders)  would  be  expected  to  yield  an  average  improvement  in  the  post-intervention  rate 
to  87%  (risk  ratio=1.5). 
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Organizational  Change 

Organizational  change  was  consistently  one  of  the  most  effective  interventions  at  increasing  use  of  clinical 
and  preventive  services.  For  immunizations,  it  was  the  best  intervention  component,  with  an  odds  ratio  of 
7.17.  Similarly,  the  odds  ratio  was  18.1  for  organizational  change  in  colon  cancer  screening  (FOBT).  Odds 
ratios  for  improving  mammography  and  cervical  smear  cytology  rates  were  2.26  and  2.65  respectively. 
Organizational  change  was  also  the  most  heterogeneous  intervention.  Table  8  contains  descriptions  of  the 
interventions  classified  as  "organizational  change"  which  were  entered  into  the  regression  models. 
Organizational  change  was  always  just  one  component  of  a  multi-faceted  intervention.  It  was  most  often  used 
in  conjunction  with  patient  reminders.  Clearly,  no  organizational  change  will  be  consistently  effective  for  all 
services  and  all  settings  (i.e.,  there  is  no  "magic  bullet). 
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Special  Interventions:  Standing  Orders 

Seven  articles  assessed  the  effects  of  standing  orders  as  an  intervention  to  improve  preventive  care  cancer 
screening  or  immunization  rates.  Three  articles  assessed  their  effects  on  influenza  vaccination  rates 
(Margolis,  1992;  Brenton,  1998;  PRO#OH025,  1996),  two  articles  assessed  their  effects  on  pneumococcal 
rates  (Klein,  1986;  Rhew.  1999).  one  article  assessed  their  effects  on  both  influenza  and  pneumococcal 
vaccination  rates  (Herman,  1994),  and  the  last  article  assessed  their  effects  on  mammography  rates  (Herman, 
1995). 

The  first  influenza-only  article  compared  usual  care  with  provider  education,  patient  education,  provider 
reminders,  and  patient  reminders  (Margolis,  1992).  The  intervention  also  included  modification  of 
organizational  policies  to  allow  nursing  staff  to  administer  influenza  vaccination  to  patients  without  a  signed 
physician  order.  There  were  two  control  sites,  each  of  which  was  paired  with  an  intervention  site.  The 
vaccination  rates  of  the  two  control  sites  did  not  change  significantly  over  the  course  of  the  study.  Only  one 
of  the  two  intervention  sites  showed  a  significant  increase  in  vaccination  rates,  from  56%  to  72%  (NNT=5 
when  compared  with  the  control  site).  The  second  influenza-only  article  compared  usual  care  with  provider 
education  and  a  number  of  organizational  changes,  including  nurse  standing  orders  (PRO#  OH025).  While 
the  percentage  of  vaccinated  patients  remained  unchanged  in  the  usual  care  group,  the  rate  in  the  intervention 
group  increased  from  15.8%  to  19.4%  (NNT=28  when  compared  with  the  control  group). 

In  the  third  influenza-only  article,  patients,  physicians,  communities,  hospitals,  and  long-term  care  facilities  in 
Iowa  were  targeted  with  a  number  of  interventions  (Brenton,  1998).  The  primary  intervention  for  the 
hospitals  was  a  standing  order  to  immunize  high-risk,  hospitalized  patients.  Even  though  the  hospitals 
administered  immunizations  to  only  7%  of  high-risk  hospitalized  patients,  immunization  rates  in  the  counties 
where  these  hospitals  were  located  increased  by  12.1  percentage  points  (from  40.6%  to  52.7%)  compared 
with  an  increase  of  8.9  percentage  points  in  the  control  counties  (from  32.8%  to  41.7%).  The  authors 
believed  that  these  hospital  programs  had  a  spillover  effect  on  care  provided  in  the  surrounding  communities. 
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In  the  first  pneumococcal-only  article,  the  effects  of  posters  containing  information  about  immunizing  high- 
risk  patients  were  compared  with  the  effects  of  posters  and  immunization  standing  orders  carried  out  by  a 
hospital  infection-control  nurse  (Klein,  1986).  None  of  the  patients  in  the  poster-only  group  received 
pneumococcal  vaccination,  compared  with  78%  in  the  poster  plus  standing  order  group. 

The  second  pneumococcal-only  article  examined  three  groups  of  interventions:  patient  and  provider 
reminders,  nurse  standing  orders  with  feedback  and  patient  and  provider  reminders,  and  nurse  standing  orders 
with  patient  and  provider  reminders  (Rhew,  1999).  Only  5%  of  the  patient  and  provider  reminder  groups 
received  pneumococcal  vaccination,  compared  with  22%  of  the  nurse  standing  orders  with  feedback  and 
patient  and  provider  reminder  group,  and  25%  of  the  nurse  standing  orders  with  patient  and  provider  reminder 
group. 

The  article  that  assessed  effects  of  standing  orders  on  both  influenza  and  pneumococcal  rates  contained  three 
intervention  groups  (Herman,  1994).  The  intervention  for  the  first  group  consisted  of  provider  education 
only.  The  second  group's  intervention  consisted  of  provider  education  and  patient  education.  The  third 
group's  intervention  consisted  of  provider  education,  patient  education,  provider  reminders,  and 
organizational  changes,  including  nursing  standing  orders.  The  pneumococcal  vaccination  rate  for  the  third 
group  increased  21.6%  compared  with  an  increase  of  5.1%  for  the  second  group  and  an  increase  of  3.4%  for 
the  first  group.  The  influenza  vaccination  rate  for  the  third  group  increased  by  55.1%,  compared  with  an 
increase  of  44.6%  for  the  second  group  and  an  increase  of  41.7%  for  the  first  group. 

The  last  article,  which  assessed  the  effects  of  standing  orders  on  mammography  rates,  contained  three 
intervention  groups.  All  three  groups  received  provider  education  (Herman,  1995).  Groups  two  and  three 
received  patient  education  as  well.  In  addition,  group  three's  intervention  also  encompassed  provider 
reminders  and  organizational  change,  including  standing  orders.  The  results  showed  that  mammography  rates 
increased  by  17%  in  group  one,  by  31%  in  group  two,  and  by  37%  in  group  three. 
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In  conclusion,  in  nearly  every  instance,  organizational  changes  that  included  standing  orders  improved  the 
rates  of  vaccination  and  mammography  when  compared  to  usual  care  or  to  other  interventions.  Since  many  of 
these  studies  had  other  interventions  besides  standing  orders,  the  effect  size  for  standing  orders  alone  is 
unclear. 

We  identified  a  number  of  additional  studies  relevant  to  Question  1  that,  for  statistical  reasons,  we  were 
unable  to  include  in  our  meta-regression  analysis.  These  studies  fall  into  five  categories: 

1.  mass  mailings  by  Peer  Review  Organizations  (PROs) 

2.  sigmoidoscopy  studies 

3.  mass  media  studies 

4.  regulatory  change  studies 

5.  miscellaneous  studies. 

We  present  a  qualitative  summary  of  each  of  these  studies  below. 
Mass  Mailings  by  peer  review  Organizations 

We  identified  five  mass  mailing  studies  performed  by  Peer  Review  Organizations  that  we  were  unable  to 
include  in  our  meta-regression  analysis  due  to  the  large  sample  size.  Each  study  was  designed  to  increase  the 
receipt  of  influenza  vaccinations  and  was  described  in  an  NPD.  Table  9  summarizes  their  key  aspects. 

The  first  study,  a  community-based  randomized  controlled  trial,  was  performed  in  Wyoming  and  Montana 
and  reported  a  modest  but  statistically  significant  improvement  in  influenza  vaccinations  among  patients  who 
received  a  personalized  letter  or  a  form  letter  relative  to  those  who  did  not.  These  results  were  published  in 
Morbidity  and  Mortality  Weekly  Report  (1995)  and  led  to  four  additional  studies  on  this  topic,  performed  by 
other  Peer  Review  Organizations.  None  of  the  other  four  studies  showed  a  clinically  important  effect  for 
mass  mailings,  regardless  of  the  number  or  type  of  mailing  (e.g.,  personalized  or  form  letter),  or  target 
population  (e.g.,  all  Medicare  beneficiaries  versus  high-risk  Medicare  beneficiaries).  We  therefore  conclude 
that  mass  mailings  have  clinically  trivial  effects  in  improving  receipt  of  influenza  vaccinations. 
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Information  on  two  additional  studies  of  mass  mailing  interventions  was  forwarded  to  us  as  this  report  was  in 
its  final  assembly.  The  first  study  was  a  controlled  before-and-after  study  to  increase  mammography  rates, 
performed  in  Delaware.  The  intervention  consisted  of  a  statewide  publicity  campaign  with  the  Lieutenant 
Governor  of  Delaware  (an  older  woman)  as  spokesperson,  followed  by  a  mass  mailing  to  female  residents  of 
one  county.  Women  in  the  intervention  county  received  a  personalized  letter  from  the  Lieutenant  Governor. 
The  mammography  rate  in  the  control  county  increased  from  30.3%  to  33.6%,  while  the  mammography  rate 
in  the  intervention  county  increased  from  23.8%  to  29.9%.  The  difference  in  the  increase  in  mammography 
rates  between  counties  just  reached  statistical  significance. 

A  similar  subsequent  study  in  West  Virginia  used  the  First  Lady  of  West  Virginia  as  the  spokesperson  and 
added  a  mass  media  campaign  to  a  mass  mailing.  Upon  preliminary  analysis,  this  study  appears  to  have  had 
greater  effectiveness  than  the  Delaware  study  (Charles  P.  Schade,  MD,  MPH,  personal  communication).  The 
increased  benefit  seen  with  this  PRO  mass  mailing  relative  to  those  reported  above  could  be  due  to  several 
factors,  including  targeting  a  different  service  (mammography  rather  than  influenza  vaccination)  and  the  use 
of  a  media  campaign  with  an  identifiable  spokesperson.  Of  the  influenza  studies  reported  above,  only  one 
(Minnesota)  employed  any  additional  interventions,  and  they  were  limited  to  provider  education  in  three 
communities. 

Sigmoidoscopy  Studies 

Five  studies  on  sigmoidoscopy  utilization  passed  our  screening  process.  This  was  too  small  a  sample  for 
calculating  pooled  effects  or  performing  a  meta-regression.  None  of  these  studies  addressed  sigmoidoscopy 
alone:  each  was  designed  to  increase  colon  cancer  screening  with  the  Fecal  Occult  Blood  Test  (FOBT)  as 
well.  Moreover,  several  of  the  studies  attempted  to  increase  utilization  of  other  preventive  services  (i.e., 
mammography  and  cervical  smear  cytology),  using  interventions  similar  to  those  for  sigmoidoscopy  and 
FOBT. 
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Table  10  summarizes  the  sigmoidoscopy  study  data.  The  third  and  fourth  columns  indicate  the  proportion  of 
each  group  who  received  sigmoidoscopy  before  and  after  the  intervention.  Several  studies  failed  to  report  key 
data,  such  as  the  sample  size  (Dietrich,  1992)  or  baseline  utilization  rates  (Bejes,  1992;  Clementz,  1990  ). 

These  studies  most  often  targeted  interventions  toward  physicians.  For  example,  three  studies  (Bejes,  1992; 
McPhee,  1989;  and  McPhee,  1991)  investigated  the  effect  of  placing  physician  reminder  sheets  in  patient 
charts.  Two  of  them  (Bejes,  1992;  McPhee,  1989)  showed  positive  and  significant  results  relative  to  the 
control  group.  The  other  study  (McPhee,  1991)  showed  a  positive,  but  not  statistically  significant,  trend. 

One  study  (Clementz,  1990)  mailed  patient  reminders  to  asymptomatic  women  age  50  to  69.  These 
individuals  received  both  a  personalized  letter  signed  by  their  primary  care  physician  and  educational 
materials  describing  their  general  risks  for  colon  cancer.  A  second  reminder  letter,  which  also  included 
patient  educational  materials,  was  mailed  four  weeks  after  the  first  letter.  Surprisingly,  only  one  percent  of 
the  intervention  group  appeared  for  screening,  compared  with  over  five  percent  of  the  control  group. 
However,  this  difference  between  the  two  groups  was  not  statistically  significant. 
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Mass  media  studies 

Due  to  the  large  population  sizes  targeted  in  media  campaigns,  these  interventions  could  not  be  entered  into 
the  meta-regression  models.  Media  campaigns  were  very  rarely  used  without  an  additional  patient-level 
intervention.  For  example,  Bennett  (1994)  reported  on  an  influenza  vaccine  media  campaign  that  was  used  in 
conjunction  with  patient  and  provider  education,  patient  and  provider  reminders,  and  a  provider  financial 
incentive.  Thus,  we  could  not  come  to  a  conclusion  about  the  effects  of  media  campaigns  per  say. 

REGULATORY  CHANGE  STUDIES 

In  1987,  amendments  to  the  Ontario  Health  Protection  and  Promotion  Act  led  to  a  legal  requirement  for 
informing  patients  of  benefits,  risks,  alternatives  to  immunization,  and  reportable  signs  and  symptoms  before 
vaccination.  In  their  1994  study,  Charles  and  Lewis  examined  whether  requiring  elderly  patients  to  sign 
consent  forms  prior  to  receiving  the  influenza  vaccine  affected  vaccination  rates.  All  patients  65  years  of 
age  and  older  who  had  attended  the  Sunnybrook  Health  Science  Centre  Family  Practice  Unit  at  the 
University  of  Toronto  within  three  years  prior  to  the  start  of  the  study  were  included  unless  the  patient  was 
suffering  with  dementia  or  the  research  team  could  not  locate  the  patient's  chart.  All  participants  received  a 
package  containing  information  from  the  Ministry  of  Health  on  the  influenza  vaccine  and  a  questionnaire 
covering  demographics  and  previous  influenza  vaccination;  subjects  assigned  to  the  intervention  group  also 
received  and  were  asked  to  sign  a  consent  form  prior  to  receiving  the  influenza  vaccination.  All  subjects  in 
the  intervention  group  who  attended  the  clinic  during  the  study  period  (n  =  52)  signed  the  consent  form, 
however,  one  subject  was  reluctant  to  do  so.  The  results  showed  that  there  was  no  statistically  significant 
difference  in  vaccination  rates  between  the  intervention  and  control  groups  suggesting  that  requiring  elderly 
patients  to  sign  consent  forms  did  not  deter  patients  from  receiving  the  influenza  vaccine. 

These  results  differ  from  another  study  (Patriarca  et  al.,  1985)  cited  by  Charles  and  Lewis  where  requiring 
signed  consent  from  relatives  of  nursing  home  residents  in  the  United  states  led  to  statistically  significant 
lower  rates  of  influenza  vaccination.  The  two  studies  differ,  however,  not  only  in  setting  but  also  in  that 
Charles  and  Lewis  required  elderly  patients  to  sign  their  own  consent  forms;  the  two  studies,  therefore,  may 
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not  be  comparable.  Consistent  with  other  studies,  Charles  and  Lewis  reported  that  vaccination  rates 
increased  significantly  when  study  participants  were  mailed  reminder  letters  and  information  packages  were 
placed  in  patient  charts. 

MISCELLANEOUS  STUDIES 

We  excluded  several  other  studies  from  our  regression  models,  mainly  because  they  did  not  report  results  at 
the  patient  level.  Although  this  method  of  reporting  may  have  been  appropriate,  particularly  if  the  unit  of 
allocation  was  the  provider  or  community,  we  could  not  include  the  results  in  our  meta-regression  analysis. 
We  describe  these  remaining  studies  briefly  below. 

Dietrich  and  colleagues  (1992)  reported  on  the  effects  of  a  day-long  physician  educational  session  and  an 
office  intervention  that  included  the  use  of  medical  record  flow  sheets  and  division  of  responsibilities  for 
providing  services-among  medical  staff.  Dietrich  found  that  the  office  system  intervention  was  associated 
with  an  increase  in  mammography  and  FOBT,  where  as  education  was  only  associated  with  an  increase  in 
mammography.  Neither  intervention  was  successful  in  significantly  increasing  the  utilization  of 
sigmoidoscopy.  These  results  are  in  general  in  agreement  with  those  of  our  meta-regression. 

Rosser  and  colleagues  (1991)  tested  computerized  reminder  systems  used  in  the  delivery  of  five  preventive 
procedures  (including  cervical  smear  cytology  and  influenza  vaccination)  in  a  family  practice.  The  patients  in 
the  first  intervention  group  received  a  telephone  call  or  letter  reminding  them  of  any  overdue  preventive 
procedures;  the  physicians  of  the  patients  in  the  second  group  were  reminded  to  provide  any  overdue  service 
at  an  office  visit.  Both  intervention  groups  had  significantly  higher  rates  of  utilization  than  did  the  control 
group.  Again,  these  results  are  in  general  in  agreement  with  those  of  our  meta-regression. 

A  third  study  involved  a  media  campaign  to  increase  the  utilization  of  cervical  smear  cytology  in  Victoria, 
Australia  (Cecchini,  1989).  The  intervention  included  sending  personalized  letters  to  women  inviting  them  in 
for  screening.  It  also  incorporated  a  general  practice  component,  one  facet  of  which  involved  face-to  face 
educational  outreach  to  the  doctor.  There  was  no  significant  difference  in  cervical  cancer  screening  patterns 
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between  the  intervention  region  and  the  control  region.  The  lack  of  effect  of  personalized  patient  reminders 
runs  counter  to  meta-regression  results. 

Mandel,  Franks,  and  Dickinson  (1985)  studied  the  effects  of  providing  educational  feedback  to  medical 
interns  about  preventive  care  deficiencies  found  in  their  charts.  The  two  procedures  relevant  to  this  report  are 
cervical  smear  cytology  and  FOBT.  The  intervention  did  not  improve  intern  performance  in  these  areas 
significantly.  The  lack  of  effect  of  feedback  is  consistent  with  our  meta-regression  results. 

We  also  excluded  a  randomized  controlled  trial  (Ward,  1996)  of  a  three-day  workshop  designed  to  increase 
trainees'  rates  of  smoking  cessation  counseling  and  cervical  smear  cytology  orders  because  it  did  not  report 
the  study's  sample  size.  The  post-workshop  difference  between  the  control  and  intervention  group  was  not 
significant,  although  the  rate  of  cervical  smear  cytology  discussions  with  patients  did  increase  slightly  in  the 
intervention  group. 

Majeed  and  colleagues  (1997)  investigated  the  use  of  reminder  letters  for  mammography  in  London  area 
clinics.  They  found  a  statistically  significant  increase  in  the  utilization  of  breast  cancer  screening  in 
intervention  practices  compared  to  control  practices.  However,  the  absolute  increase  in  uptake  (from  53.8% 
to  58.5%)  was  small  and  of  limited  public  health  importance.  This  limited  effect  of  patient  reminders  is 
somewhat  counter  to  the  results  from  our  meta-regression. 

Tierney,  Hui,  and  McDonald  (1986)  studied  the  effects  of  monthly  feedback  reports,  and  /or  of  specific 
reminders  given  to  physicians  at  the  time  of  patient  visits.  Services  studied  included  FOBT,  mammography, 
and  pneumococcal  vaccination.  Again,  results  were  reported  at  the  physician  level.  Staff  receiving  feedback 
more  often  ordered  these  three  services  than  did  controls.  Utilization  levels  for  physicians  receiving  both 
feedback  and  chart  reminders  was  twice  that  of  physicians  receiving  feedback  alone.  The  positive  effect  of 
feedback  is  counter  to  our  meta-regression  results,  though  the  positive  effect  of  physician  reminders  is 
consistent  with  our  results. 
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In  1987  RAND  published  findings  from  its  Health  Insurance  Experiment,  which  was  conducted  from  1974  to 
1982.  Free  care  was  found  to  have  some  benefit  for  the  three  screenings  for  early  detection  of  cancer, 
compared  to  plans  where  costs  were  shared.  For  example,  65%  of  women  aged  45-65  on  the  free  plan  had 
cervical  smear  cytology  in  the  initial  three  year  period,  compared  with  5 1 .9%  in  the  cost-sharing  plans.  The 
positive  effect  of  patient  financial  incentives  agrees  with  the  results  of  our  meta-regression  analysis. 

Question  2:  relative  effectiveness  of  each  intervention 

What  is  the  relative  effectiveness  of  each  intervention  in  improving  the  use  of  clinical  preventive  and 
screening  services? 

We  considered  two  forms  of  evidence:  direct  evidence  and  indirect  evidence.  Direct  evidence  consisted  of 
head-to-head  comparisons  of  single  interventions.  Indirect  evidence  consisted  of  the  magnitude  of  the  odds 
ratios  from  the  main  meta-regression.  (For  example,  in  Table  6  for  immunizations,  organizational  change  had 
an  adjusted  odds  ratio  of  7.17,  while  provider  reminder  had  an  odds  ratio  of  1.66.)  We  identified  nine  studies 
that  assessed  a  single  intervention  relative  to  another  single  intervention.  Of  these  nine,  one  was  a  mass 
mailing  study,  previously  described  above.  Of  the  remaining  eight,  four  of  them  directly  confirmed  the  order 
of  relative  effectiveness  seen  in  the  meta-regression  analysis.  Specifically,  a  study  of  mammography  (Landis, 
1992)  and  a  study  of  cervical  smear  cytology  (Binstock,  1997)  both  found  that  patient  reminders  were 
superior  to  provider  reminders;  a  study  of  fecal  occult  blood  testing  (Thompson,  1986)  found  that  patient 
reminders  were  superior  to  patient  education;  and  a  study  of  several  preventive  services  (Tierney,  1986)  found 
that  reminders  were  superior  to  feedback. 

Two  more  studies  demonstrated  a  trend  similar  to  that  seen  in  our  meta-regressions.  One  study  of 

mammography  (Sharp,  1996)  found  that  patients  receiving  reminders  underwent  more  mammograms  than  did 

patients  receiving  a  home  visit  from  a  nurse  with  or  without  patient  education,  but  this  difference  was  not 

statistically  significant.  Similarly,  a  study  of  cervical  smear  cytology  (Pierce,  1989)  also  found  that  patients 

receiving  reminders  underwent  more  cervical  smear  cytology  than  patients  of  doctors  who  received  provider 

reminders,  but  again  this  difference  was  not  statistically  significant. 
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The  last  two  studies  showed  an  order  of  effectiveness  opposite  to  that  seen  in  our  meta-regressions.  In  the 
first  one  (Moran,  1996),  patient  education  was  superior  to  a  patient  financial  incentive.  However,  in  this  case 
the  financial  incentive  was  relatively  weak:  i.e.,  if  a  patient  received  an  influenza  vaccine,  he  or  she  became 
eligible  for  a  lottery  to  receive  a  $50  voucher  for  groceries.  In  the  second  study  (Mant,  1992),  which 
addressed  fecal  occult  blood  testing,  patient  reminders  were  superior  to  organizational  change  which,  in  this 
case,  consisted  of  having  a  nurse  offer  patients  a  hemoccult  test  during  a  health  check  up.  In  general,  though, 
the  direct  evidence  from  the  few  studies  that  directly  compared  interventions  supported  the  order  of  relative 
effectiveness  observed  in  our  meta-regression  analyses. 

EFFECTIVENESS  OF  MULTIPLE  INTERVENTIONS 

In  addition  to  these  direct,  one-on-one  comparisons  of  interventions,  we  also  examined  whether  multiple 
interventions  produced  better  results  than  single  interventions.  Direct  evidence  consisted  of  studies  that 
directly  compared  a  single  intervention  to  that  same  intervention  in  conjunction  with  additional  interventions. 
Indirect  evidence  consisted  of  the  results  from  a  meta-regression  analysis  in  which  we  compared  the 
effectiveness  of  a  single  intervention  versus  a  control  or  usual  care  group  with  the  effectiveness  of  multiple 
interventions  against  a  control  or  usual  care  group  for  the  same  preventive  service.  We  can  compare  the 
marginal  effectiveness  of  adding  an  intervention  component  to  the  "average"  single  component  intervention 
across  our  studies.  Because  of  the  paucity  of  direct  evidence,  we  will  discuss  the  indirect  evidence  for  these 
comparisons  first. 

Table  1 1  presents  the  results  of  our  meta-regression  analysis  comparing  the  adjusted  odds  ratios  of  single 
interventions  to  multiple  interventions  for  each  service.  There  was  no  statistical  difference  between  the  odds 
ratios  for  single  interventions  versus  multiple  interventions  for  either  immunizations  or  mammography, 
indicating  that  multiple  interventions  did  not  achieve  greater  effectiveness  than  did  single  interventions, 
relative  to  control  or  usual  care.  For  cervical  smear  cytology,  however,  the  difference  between  the  average 
adjusted  odd  ratios  for  single  interventions  and  for  multiple  interventions  was  statistically  different,  with  a 
marginal  odds  ratio  of  1.25  (95%  confidence  interval:  1.14  to  1.37).  We  note  that  as  expected,  this  marginal 
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odds  ratio  is  the  ratio  of  individual  odds  ratios  versus  control  (1.98/1.58=1.25).  This  result  indicated  that  for 
cervical  smear  cytology,  multiple  interventions  did  achieve  more  benefit  than  single  interventions  alone,  but 
the  effects  were  not  additive  (i.e.,  the  incremental  effects  diminished  as  additional  interventions  were  added). 
Similarly,  the  difference  between  the  average  adjusted  odds  ratios  for  colon  cancer  was  also  statistically 
significant,  producing  a  marginal  odds  ratio  of  2.50  (95%  confidence  interval:  2.08  to  3.00).  This  result 
indicated,  however,  that  for  colon  cancer,  interventions  with  multiple  components  achieved  much  more 
effectiveness  than  did  interventions  with  single  components,  and  that  the  effect  was  additive. 
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Table  12  presents  each  intervention,  by  preventive  service,  that  had  a  statistically  significant  marginal 
adjusted  odds  ratio  greater  than  one.  These  are  the  only  interventions  that,  on  average,  would  be  expected  to 
produce  statistically  significant  improvements  when  added  to  an  "average"  intervention  across  studies  which 
did  not  contain  that  particular  component. 
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Our  interpretation  of  the  direct  evidence  for  the  effectiveness  of  single  versus  multiple  interventions  was 
again  limited  by  lack  of  data.  We  did  identify  32  studies  that  compared  a  single  intervention  to  that  same 
intervention  plus  a  second  intervention,  generating  a  total  of  39  comparisons.  Among  these  39  comparisons, 
24  favored  multiple  interventions,  13  showed  no  difference,  and  two  favored  the  single  intervention.  The 
interpretation  of  these  results,  however,  is  complex  because  interventions  vary  considerably  in  their 
effectiveness  and  intensity  is  hard  to  measure. 

To  better  understand  the  interactions  among  different  interventions,  we  classified  the  interventions  as  either 
"strong"  (e.g.,  organizational  change,  financial  incentives,  patient  reminders,  and  provider  reminders)  or 
"weak"  (e.g.,  feedback,  patient  education,  and  provider  education).  Using  this  classification  system,  we 
predicted  that  adding  a  strong  intervention  to  either  another  strong  one  or  to  a  weak  one  would  produce 
significant,  positive  effects;  that  adding  a  weak  intervention  to  a  strong  one  would  produce  little  effect;  and 
that  adding  a  weak  intervention  to  another  weak  one  would  produce  variable  effects.  The  results  of  our 
analysis  supported  these  predictions.  All  1 1  comparisons  that  added  a  strong  intervention  to  another  strong 
intervention  reported  significant  benefits.  Similarly,  five  of  the  seven  comparisons  that  added  a  strong 
intervention  to  a  weak  one  also  reported  significant  benefits.  As  expected,  eight  of  the  ten  comparisons  that 
added  a  weak  intervention  to  a  strong  one  reported  no  benefit,  while  two  reported  some  benefit.  Finally,  of 
the  two  comparisons  in  which  a  weak  intervention  was  added  to  another  weak  intervention,  one  reported  a 
benefit  and  one  did  not. 
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Question  3:  important  covariates 

How  do  important  covariates,  such  as  the  target  population  (e.g.,  low-income  or  other  vulnerable 
populations)  or  the  setting  in  which  the  intervention  is  applied  (e.g.,  academic  versus  non-academic  practice, 
managed  care  versus  fee-for-service  systems),  impact  the  effectiveness  of  the  interventions? 

As  mentioned  above,  in  assessing  the  degree  to  which  important  covariates  (e.g.,  target  population, 
intervention  setting)  impact  the  effectiveness  of  interventions,  we  considered  both  direct  and  indirect 
evidence.  We  defined  direct  evidence  as  a  head-to-head  comparison  of  the  intervention  in  a  study  of  eligible 
design,  such  as  a  clinical  trial  or  controlled  before  and  after  study.  We  defined  indirect  evidence  as  the  results 
of  a  meta-regression  analysis  examining  the  covariates  of  interest  in  a  covariate  by  treatment  interaction. 

We  were  unable  to  identify  any  studies  that  met  our  criteria  for  direct  evidence;  furthermore,  there  was 
insufficient  data  to  include  the  covariates  in  a  covariate  by  treatment  interaction  in  our  meta-regression  model. 
Therefore,  we  included  the  covariates  as  single  variables  in  a  regression  model  across  all  interventions  for 
each  preventive  or  screening  service.  Even  with  this  adjustment,  we  were  unable  to  assess  each  variable  for 
each  service,  due  to  insufficient  data.  Table  13  presents  the  results  for  those  variables  that  we  could  assess. 
In  this  table  we  present  the  marginal  odds  ratio  for  the  "average"  intervention  applied  in  various  settings  as 
compared  with  the  "average"  intervention  applied  elsewhere.  For  example,  the  odds  of  receiving 
mammogram  screening  if  the  "average"  intervention  is  applied  to  low-income  populations  is  0.82  times  the 
odds  for  non-low-income  populations,  and  this  decrease  in  odds  is  statistically  significant  as  the  95% 
confidence  interval  does  not  cross  one.  Unfortunately,  these  results  do  not  seem  to  produce  any  consistent 
pattern.  Hence,  we  do  not  believe  firm  conclusions  can  be  drawn  regarding  the  effectiveness  of  these 
interventions  across  patient  populations  or  service  delivery  settings. 


92 


Table  13.  Important  Covariates 


Marginal  Odd  Ratio 
(95%  Confidence  Interval) 

Immunizations 

Mammography 

Cervical  Smear 
Cytology 

Colon  Cancer 
Screening  (FOBT) 

Low-income 
Populations 

— 

0.82 
(0.70-0.95) 

— 

— 

African 

American 

Populations 

— 

— 

— 

— 

Hispanic 
PoDulations 

Other  Minority 
Ponulations 

HMO  settings* 

0.95 
(0.84-  1.08) 

0.72 
(0.65  -0.80) 

0.45 
(0  37  -  0  55) 

Non-academic 
settings** 

0.51 
(0.46-0.57) 

1.02 
(0.91  -  1.16) 

1.37 
(1.21-1.54) 

2.65 
(2.18-3.23) 

Rural 

setting*** 

0.79 
(0.63  -  1.00) 

*   HMO  compared  to  fee-for-service  setting 


**   Non-academic  compared  to  academic  healthcare  setting 
***   Rural  compared  to  urban  or  suburban 
—   Insufficient  data  for  meta-analysis 

VULNERABLE  POPULATIONS 

Below,  we  provide  a  qualitative  summary  for  studies  involving  the  following  vulnerable  populations: 

•  African-Americans 

•  Hispanic  populations 

•  Native  Americans 

•  Rural  populations. 

We  did  not  identify  any  eligible  studies  specifically  addressing  the  following  vulnerable  populations: 

•  Other  minority  populations 

•  Nursing  home  populations 

•  Persons  85  and  older. 
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African-Americans 

There  were  19  articles  on  interventions  targeted  toward  improving  service  delivery  to  African  Americans. 
These  involved  42  interventions  on  flu  vaccine,  14  on  mammography,  and  9  on  cervical  smear  cytology,  and 
one  article  on  pneumococcal  vaccination.  We  found  no  interventions  designed  to  increase  use  of  colon  cancer 
screening.  Thirty-seven  interventions  used  patient  education,  36  used  patient  reminders,  28  used  provider 
education,  28  used  mass  media,  24  used  organizational  change,  19  used  provider  reminders,  eight  used 
detailing,  seven  used  patient  financial  incentives,  four  used  financial  incentives  for  organizations,  and  one 
used  audit  and  feedback.  These  groups  are  not  mutually  exclusive;  many  studies  looked  at  several  services  or 
involved  multiple  types  of  interventions  simultaneously.  The  most  effective  interventions  are  described 
below. 

In  Texas  and  Mississippi,  Peer  Review  Organizations  worked  with  Historically  Black  Colleges  and 
Universities  (HBCUs)  in  coalitions  to  improve  the  utilization  of  influenza  vaccine  among  African  Americans. 
These  interventions  included  working  with  churches,  free  clinics,  and  local  community-based  organizations  to 
raise  awareness  about  the  influenza  vaccine.  A  telephone  survey  and  focus  groups  were  used  to  identify 
barriers  and  misconceptions  about  the  vaccine.  The  Texas  Medical  Foundation  reported  an  increase  in 
utilization  of  the  flu  vaccine  by  over  10%  after  intervention;  the  Mississippi  Foundation  for  Medical  care 
reported  increases  as  high  as  23%  in  influenza  vaccine  utilization  after  intervention. 

In  two  eastern  North  Carolina  communities,  Fletcher  et  al.  (1993)  conducted  a  controlled  intervention 
program  to  increase  the  use  of  mammography  screening  for  breast  cancer.  The  intervention  consisted  of  a 
provider  education  component,  provider  reminders  (including  a  simple  prompting  system  and  periodic 
newsletters),  patient  financial  incentives  in  the  form  of  coupons  for  free  or  half  price  mammograms,  and  a 
mass  media  campaign  using  newspaper,  radio,  television,  and  community  speakers.  In  particular,  African 
American  women  were  targeted  for  intervention  by  a  Minority  Task  Force  that  was  chaired  by  an  African 
American  radio  commentator  and  included  six  African  American  women  well-known  in  the  community.  The 
experimental  community  showed  a  20%  increase  of  women  reporting  having  undergone  a  mammogram  in  the 
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previous  year,  compared  with  less  than  a  10%  increase  in  the  control  community.  The  authors  reported, 
however,  that  there  was  less  of  an  increase  for  African  American  women  than  for  white  women. 

Mandelblatt  et  al.  (1993)  conducted  a  quasi-experimental  study  at  two  urban  public  hospital  primary  care 
clinics  in  New  York  City  to  increase  screening  rates  for  cervical  and  breast  cancers.  Approximately  80%  of 
participants  at  the  control  site  and  over  90%  of  participants  at  the  intervention  site  were  African  American. 
Prior  to  the  start  of  the  study,  and  independent  from  the  study  intervention,  the  control  hospital  had  already 
implemented  a  checklist  summarizing  the  health  maintenance  protocol  that  was  attached  to  all  charts  as  a 
reminder  to  providers.  At  the  intervention  hospital,  eligible  elderly  women  arriving  for  routine  visits  were 
approached  by  nurse  practitioners  and  offered  screening.  By  the  conclusion  of  the  study,  the  intervention 
group  showed  a  significant  increase  in  screening  rates  compared  to  the  control  group.  At  the  intervention 
hospital,  the  annual  rate  of  cervical  smears  increased  from  a  baseline  rate  of  17.8%  to  56.9%; 
mammographies  increased  from  18.3%  to  40.0%.  By  comparison,  at  the  control  hospital  cervical  smear  rates 
increased  from  1 1.8%  to  18.2%,  while  rates  of  mammographies  showed  no  increase. 

In  a  randomized  controlled  trial.  Sung  et  al.  (1997)  implemented  an  intervention  to  increase  cancer  screening 
rates  among  inner-city  African  American  women.  Participants  were  recruited  through  a  variety  of  local 
community-based  organizations  catering  to  inner-city,  African  American  women.  The  intervention  group 
received  multiple  visits  from  a  lay  health  worker  (LHW),  who  had  been  recruited  from  the  National  Black 
Women's  Health  Project.  The  LHW  visited  subjects  in  their  homes  and  provided  information  on  breast  and 
cervical  cancer  and  their  screening  procedures.  All  materials  were  presented  from  a  culturally  relevant 
perspective.  After  the  intervention,  the  slight  increases  in  cervical  smear  rates  were  similar  among  both  the 
control  and  intervention  subjects.  Mammography  rates  in  the  intervention  group,  however,  were  three  times 
that  observed  in  the  control  group,  resulting  in  a  9.8%  difference  in  change  (p  <0.05)  between  the  two  groups. 

An  article  appeared  just  as  this  document  was  going  to  press  that  reported  the  results  of  a  randomized  clinical 
trial  of  a  low-literacy  patient  education  tool  on  pneumococcal  vaccination  rates  (Jacobson,  1999).  The  study 

95 


was  conducted  at  an  urban  public  teaching  hospital  serving  a  predominately  indigent,  low-literate,  African- 
American  population.  Among  221  persons  receiving  the  intervention,  20%  received  the  vaccination, 
compared  to  4%  of  212  persons  who  received  an  educational  brochure  about  nutrition  (p<0.001). 

Hispanic  populations. 

We  identified  four  studies  of  interventions  targeted  to  improve  cancer  screening  among  Hispanics;  however, 
we  were  unable  to  identify  any  studies  targeted  to  improve  vaccination  rates. 

The  first  study  (Suarez,  1997)  was  a  three-year  community-based  intervention  implemented  at  two  sites  in 
Texas.  The  goal  was  to  increase  the  use  of  mammography  and  cervical  smear  cytology.  The  intervention 
community  received  a  variety  of  mass  media  campaigns,  the  use  of  culturally  appropriate  role  models  to 
deliver  educational  messages,  and  outreach  activities  by  the  local  public  health  department.  Across  both 
services  and  all  age  groups,  the  intervention  failed  to  have  a  statistically  significant  effect.  However,  the 
overall  secular  trend  was  toward  an  increase  in  screening  rates  in  both  communities,  for  both  services. 
Among  the  subgroup  of  women  age  65  or  older,  the  intervention  did  have  a  positive  effect  on  use  of  cervical 
smear  cytology  (difference  in  intervention  versus  control  community,  +1 1.2  percentage  points),  but  a  negative 
effect  on  the  use  of  mammography  (difference  in  intervention  versus  control  community,  -14.3  percentage 
points).  The  authors  hypothesized  that  the  failure  of  their  intervention  was  due  to  lack  of  focus  and 
contamination  of  their  intervention  by  other  programs. 

The  second  study  (PRO#  FL68E052,  1998)  attempted  to  increase  mammography  rates  in  five  counties  in 
south  Florida  by  using  community-based  interventions.  Three  communities  received  the  following 
interventions: 

1.  patient  education  (i.e.,  letters  addressing  the  potential  barriers  to  and  myths  about  mammography  were 
sent  to  Hispanic  women), 

2.  provider  education  (e.g.,  letters  were  sent  to  doctors  urging  them  to  educate  their  patients  about  the 
importance  of  mammography,  and  physician-written  articles  on  the  importance  of  mammography  were 
distributed  through  a  variety  of  channels),  and 
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3.    a  wide-ranging  mass  media  campaign. 

One  of  the  three  intervention  counties  experienced  a  statistically  significant  increase  of  10  percentage  points 
(from  77%  to  87%)  in  the  two-year  rate  of  use  of  mammography.  There  was  no  statistically  significant 
change  in  the  use  of  mammography  in  any  of  the  other  counties,  and  no  change  in  the  combined  measure  of 
effect  of  the  interventions. 

In  the  third  study,  Navarro  and  colleagues  (1995)  reported  on  the  "Por  La  Vida"  intervention  program  in  San 
Diego,  CA.  This  program  employed  lay  community  workers  ("consejeras")  to  conduct  educational  group 
sessions  aimed  at  increasing  utilization  of  cervical  smear  cytology  and  mammography  among  Latinas.  Data 
collected  at  baseline  suggested  that  lack  of  knowledge,  costs  of  cancer  screening  tests,  and  lack  of  a  regular 
health  care  provider  were  major  obstacles  to  utilization.  Subjects  who  attended  a  multi-session  group 
program  on  community  living  skills  served  as  a  control.  Utilization  of  cervical  smear  cytology  increased 
from  80.4%  to  93.5%  in  the  intervention  group,  while  the  percentage  of  women  over  age  40  who  reported 
ever  having  a  mammogram  rose  from  51.8%  to  64.3%. 

The  fourth  study  (Manfredi,  1998)  evaluated  an  HMO-sponsored  intervention  to  improve  utilization  of 
cervical  smear  cytology,  mammography,  and  FOBT  in  private  physician  practices  located  in  African- 
American  and  Hispanic  Chicago  neighborhoods.  The  multi-faceted  intervention  included  an  office  chart 
reminder  system,  patient  health  maintenance  card,  on-site  training,  a  physician  continuing  medical  education 
seminar,  and  quality  assurance  visits  with  feedback  to  physicians.  The  intervention  was  successful  in 
increasing  utilization  of  FOBT  and  cervical  smear  cytology,  but  not  at  increasing  mammography  rates. 
Results  were  not  reported  by  race/  ethnicity. 

Native  Americans. 

One  study  (Dignan,  1996)  reported  on  a  five  year,  National  Cancer  Institute-funded  trial  of  health  education 
designed  to  increase  the  use  of  cervical  smear  cytology  by  Native  American  women  in  North  Carolina.  The 
intervention  was  an  individualized  health  education  program  delivered  by  female  Cherokee  lay  persons. 
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Social  learning  theory  provided  the  foundation  for  the  program,  which  also  incorporated  the  health  belief 
model,  the  minority  health  communication  model  and  the  communication-behavior  change  model.  At  post 
test,  73.2%  of  the  women  who  received  the  education  program  reported  having  received  a  cervical  smear 
cytology  following  the  intervention,  compared  with  64.0%  of  the  control  subjects. 

Rural  populations 

Interventions  in  rural  areas  were  less  successful  than  those  focusing  on  urban  or  suburban  populations.  There 
were  17  articles  on  interventions  in  rural  areas.  These  articles  described  four  interventions  on  influenza 
vaccine,  nine  on  mammography,  six  on  cervical  smear  cytology  and  five  on  colon  cancer  screening.  We 
found  no  interventions  designed  to  increase  use  of  pneumococcal  vaccine.  Seven  interventions  used  provider 
education;  ten  used  patient  education,  twelve  used  provider  reminders,  fourteen  used  patient  reminders,  seven 
used  organizational  change,  and  five  used  patient  financial  incentives.  (These  groups  are  not  mutually 
exclusive;  many  studies  involved  several  services  or  multiple  types  of  interventions.)  Patient  financial 
incentives  were  the  most  successful  interventions  with  rural  populations.  Several  examples  are  described 
below. 

As  part  of  a  demonstration  project  in  five  rural  counties  in  northwest  Pennsylvania,  a  group  of  Medicare 
beneficiaries  completed  a  health  risk  appraisal  and  were  randomized  to  three  groups.  The  first  group  served 
as  a  control;  the  other  participants  were  mailed  a  voucher  redeemable  for  a  free  flu  shot  either  from  a 
community  hospital  (Group  2)  or  primary  care  physician  (Group  3).  Immunization  rates  rose  significantly 
from  41.2%  to  63.6%  in  Group  2  and  from  41.3%  to  69.1%  in  Group  3. 

Stoner  and  colleagues  (1998)  reported  on  a  voucher  intervention  which  took  place  in  two  farming 

communities  in  southern  Minnesota.  A  90  minute  seminar  to  educate  local  physicians  on  mammography 

techniques  and  guidelines  was  held  in  each  county.  A  community  mailing  with  information  about  risk 

factors,  screening  techniques  and  the  location  of  nearby  mammography  facilities  was  initiated  in  each  count 

In  one  county  women  received  vouchers  for  a  free  mammogram  in  their  mailed  packets.  In  the  voucher 

county,  mammography  screening  rates  increased  from  62%  to  76%  . 
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Plaskon  and  Fadden  (1995)  studied  patients  in  a  poor  rural  family  practice  in  a  geographic  area  known  for 
high  rates  of  colorectal  cancer.  All  patients  in  the  study  received  a  talk  by  their  physician  on  the  symptoms, 
prevention,  and  treatment  of  colon  cancer.  Upon  leaving  the  office,  persons  in  the  experimental  group 
received  educational  materials  and  a  free  FOBT  kit  from  the  receptionist,  while  the  control  group  received 
only  the  educational  materials  and  information  on  how  to  request  an  FOBT  kit  from  a  clinic  or  pharmacy.  In 
the  following  year,  5 1  %  of  the  experimental  group  self-reported  having  utilized  FOBT  compared  to  none  of 
the  control  group.  Still,  it  is  not  clear  whether  removing  the  financial  barrier  led  to  increased  utilization,  as 
removing  the  need  to  ask  for  an  "embarrassing"  item  may  have  been  the  main  factor. 

REIMBURSEMENT  SYSTEMS 
HMO  versus  Fee  for  Service 

The  majority  of  studies  took  place  in  mixed  reimbursement  settings.  We  did  not  find  any  studies  that  directly 
compared  fee  for  service  (FFS)  with  health  maintenance  organizations  (HMOs).  The  number  needed  to  treat 
(NNT)  ranged  from  6  to  250  for  FFS  interventions  and  from  4  to  100  in  HMO  interventions,  with  the 
exception  of  two  interventions  in  FFS  studies  and  two  interventions  in  HMO  studies  which  had  negative 
NNTs.  Examples  of  successful  HMO  interventions  are  described  in  the  next  paragraph. 

One  HMO  study  (Cherkin,  1990)  evaluated  the  effectiveness  of  patient  financial  incentives  as  a  single 
intervention.  The  intervention  group  showed  no  clinically  meaningful  increase  in  cervical  smear  cytology 
rates.  In  fact,  the  control  group  had  an  increase  of  approximately  five  percentage  points  over  their  baseline 
rate  while  the  intervention  group  had  an  increase  of  less  than  two  percentage  points.  Another  HMO  study 
(Trock,  1993)  found  a  significant  increase  in  mammography  utilization  through  the  use  of  provider  and 
patient  education  and  patient  reminders.  The  control  group  increased  utilization  from  39%  to  49%  while  the 
intervention  group  increased  utilization  from  41%  to  68%. 

A  majority  of  the  remaining  HMO  studies  did  not  report  baseline  rates,  thus  making  it  difficult  to  draw  any 
meaningful  conclusions  for  the  HMO  population. 
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Question  4:  Cost  effectiveness 

What  is  the  relative  cost  effectiveness  of  each  intervention  in  increasing  the  use  of  clinical  preventive  and 
screening  services? 

Few  studies  that  met  our  screening  criteria  provided  data  on  the  costs  of  interventions;  even  fewer  provided 
cost-effectiveness  data.  Because  of  the  paucity  of  relevant  information,  we  were  unable  to  perform 
quantitative  analyses  of  the  relative  cost-effectiveness  of  the  various  interventions.  Instead,  we  summarize 
the  results  of  these  studies  in  narrative  format  below,  along  with  data  presented  in  Table  14.  A  number  of 
studies  reported  results  on  more  than  one  condition. 

Influenza  and  pneumococcal  vaccinations 

Twelve  studies  on  improving  influenza  or  pneumococcal  vaccination  rates  provided  cost  data.  However,  only 
four  of  them  contained  cost-effectiveness  analyses.  Two  of  these  four  studies  (Rosser,  1991;  McDowell, 
1986)  examined  the  cost-effectiveness  of  computerized  physician  reminders,  patient  letter  reminders,  and 
patient  telephone  reminders.  Both  concluded  that  computerized  physician  reminders  were  the  most  cost- 
effective  of  the  three  interventions.  The  two  studies  reached  opposite  conclusions  on  the  cost-effectiveness  of 
telephone  reminders  versus  letter  reminders.  The  third  cost-effectiveness  study  (Morrissey,  1995)  reported 
that  full  Medicare  reimbursement  for  preventive  care  services,  in  combination  with  provider  reminders  and  an 
organizational  change  that  enabled  nurses  to  perform  many  preventive  care  functions,  resulted  in  Medicare 
charges  that  were  lower  than  average.  Hospital  admissions  and  lengths  of  stay  also  decreased  relative  to  usual 
care.  The  fourth  study  (Beck,  1997)  reported  that  elderly  HMO  patients  who  attended  a  cooperative  health 
care  clinic  where  influenza  and  pneumococcal  vaccinations  were  routinely  administered  experienced  fewer 
hospitalizations  and  incurred  lower  skilled  nursing  facility  costs  than  control  subjects.  The  direct  cost  savings 
of  this  health  care  program  was  $14.79  per  patient  per  month. 

One  study  reporting  the  effectiveness  of  mailed  patient  reminders  (Nexoe,  1997)  found  that  vaccination  rates 
were  higher  when  the  reminders  were  used  in  conjunction  with  free  or  low  cost  vaccinations.  A  different 
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study  (Lave,  1996)  also  found  that  waiving  fees  for  preventive  care  services  produced  an  increase  in  influenza 
vaccination  rates. 

Several  of  the  remaining  studies  provided  cost  information  on  various  interventions  used  to  increase 
immunization  rates.  For  example,  the  physicians  in  one  study  (Kouides,  1998)  were  paid  $0.80  per  influenza 
vaccination  if  their  vaccination  rate  was  >  70%  and  $  1 .60  per  vaccination  if  their  rate  was  >  85%.  This 
financial  incentive,  combined  with  frequent  feedback,  resulted  in  immunization  rates  that  were  statistically 
significantly  higher  than  control.  Another  study  (Belcher,  1990)  reported  that  the  cost  of  influenza 
vaccination  offered  through  a  health  promotion  clinic  was  $11  per  patient,  while  a  different  study 
(Hutchinson,  1989)  reported  that  a  computerized  provider  reminder  system  incurred  $15,200  Canadian  in 
start-up  costs.  In  addition,  two  studies  provided  cost  information  on  the  use  of  patient  reminders  to  increase 
immunization  rates:  one  (Smith,  1999)  reported  that  the  cost  per  mailed  patient  reminder  was  $1.26,  while 
the  other  (Buchner,  1987)  reported  direct  costs  of  $0.30  per  reminder. 

Mammography 

Eleven  studies  on  improving  mammography  rates  also  provided  cost  data.  Six  of  them  were  cost- 
effectiveness  studies;  these  generally  characterized  cost  effectiveness  in  terms  of  cost-per-additional 
mammogram  or  cost  per  additional  woman  screened.  For  example,  the  first  cost-effectiveness  study  (Atri, 
1997)  reported  that  provider  education,  patient  reminders,  and  organizational  change  cost  £13  per  additional 
woman  screened.  The  second  cost-effectiveness  study  (King,  1994),  which  provided  free  mammograms, 
reported  the  following  costs  per  additional  woman  screened: 

1 .  $0.9 1  for  a  patient  reminder  letter, 

2.  $2.73  for  a  second  reminder  letter, 

3.  $3.68  for  a  letter  suggesting  a  preventive  office  visit,  and 

4.  $4.92  for  a  patient  reminder  phone  call. 

The  third  cost-effectiveness  study  (Mohler,  1995),  which  also  examined  the  effectiveness  of  different  forms 

of  patient  remainders,  found  that,  in  terms  of  cost  per  additional  mammogram,  the  most  cost  effective  patient 

101 


reminder  was  a  phone  call  by  a  medical  assistant  ($3.00).  followed  by  a  reminder  letter  ($13.57),  and  then  a 
physician  phone  call  ($51.82).  Another  study  (Majeed,  1997)  that  employed  reminder  letters  targeted  to 
patients  in  need  of  a  mammogram  reported  that  the  marginal  cost  of  each  additional  woman  screened  was  £7. 

Another  cost-effectiveness  study  (Stevens,  1997)  found  that  academic  detailing  was  both  expensive  and 
ineffective:  although  the  intervention  cost  $34  Australian  per  physician  visited,  it  produced  no  difference  in 
mammography  rates  relative  to  usual  care.  The  final  cost-effectiveness  study  (Morrissey,  1995)  reported  that 
full  Medicare  reimbursement  for  preventive  care  services,  in  combination  with  provider  reminders  and  an 
organizational  change  that  enabled  nurses  to  perform  many  preventive  care  functions,  resulted  in  Medicare 
charges  that  were  lower  than  average.  Hospital  admissions  and  lengths  of  stay  also  decreased  relative  to  usual 
care. 

Several  of  the  remaining  studies  on  mammography  reported  only  cost  data.  For  example,  one  study  (Frame, 
1994)  found  that  the  cost  of  patient  reminders  was  over  twice  that  of  provider  reminders.  Another  study 
(Brady,  1988)  reported  that  the  cost  of  the  self-audit  portion  of  a  provider  audit  and  feedback  intervention  was 
about  equal  to  the  cost  of  the  feedback  portion  of  the  intervention.  According  to  a  different  article  (Lurie, 
1987),  undergoing  a  pap  smear  and  mammogram  every  three  years  incurs,  on  average,  $97  in  insurance  costs. 

One  of  the  last  two  studies  (Lane,  1991)  reported  no  differences  among  mammography  rates  in  areas 
providing  free  mammography,  low-cost  mammography,  and  usual-cost  mammography.  The  final  study 
(Burton,  1997)  found  that  Part  A  Medicare  claims  for  patients  receiving  an  intervention  of  patient  financial 
incentives,  provider  financial  incentives,  and  patient  reminders  were  less  than  those  of  the  usual  care  group. 
However,  there  were  no  differences  in  Part  B  claims,  and  a  regression  model  showed  no  differences  in  Parts 
A  or  B  claims  during  the  third  and  fourth  years  after  the  intervention. 

Cervical  Smear  Cytology 

Eleven  articles  on  improving  cervical  smear  cytology  rates  also  provided  cost  data.  However,  only  four  of 
these  provided  cost-effectiveness  results.  One  of  these  (McDowell.  1989),  which  reported  costs-per- 
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additional  cervical  smear  obtained,  found  that  computerized  physician  reminders  were  more  cost-effective 
than  nurse  telephone  reminders  to  patients,  which  were  more  effective  than  reminder  letters  to  patients.  A 
subsequent  update  of  this  study  (Rosser,  1991)  confirmed  that  computerized  physician  reminders  were  the 
most  cost-effective  of  the  three  interventions;  however,  the  update  found  that  patient  letter  reminders  were 
more  cost  effective  than  telephone  reminders.  A  third  study  (Binstock,  1997),  which  also  reported  cost  per 
additional  cervical  smear  obtained,  ranked  interventions  in  the  following  order,  from  most  cost  effective  to 
least  cost  effective:  computerized  provider  reminder  ($2.99),  patient  reminder  letter  ($4.76),  patient 
telephone  reminder  ($7.99),  and  provider  memo  listing  patients  in  need  of  a  cervical  smear  ($22.96).  The 
final  cost-effectiveness  study  (Morrissey,  1995)  reported  that  full  Medicare  reimbursement  for  preventive  care 
services,  in  combination  with  provider  reminders  and  an  organizational  change  that  enabled  nurses  to  perform 
many  preventive  care  functions,  resulted  in  Medicare  charges  that  were  lower  than  average.  Hospital 
admissions  and  lengths  of  stay  also  decreased  relative  to  usual  care. 

Several  of  the  remaining  studies  on  cervical  smear  cytology  reported  only  cost  data.  For  example,  one  study 
(Frame,  1994)  found  that  the  cost  of  patient  reminders  was  over  twice  that  of  provider  reminders,  while 
another  study  (Stevens,  1997)  reported  that  the  cost  of  academic  detailing  was  $34  Australian  per  physician 
visited.  A  third  study  (Robson,  1989)  reported  that  a  computer  system  that  enabled  a  health  promotion  nurse 
and  physician  to  assess  health  risk  factors  incurred  £12,000  in  start-up  costs  and  £2,000  in  maintenance  costs 
per  year.  A  fourth  study  (Lurie,  1987)  reported  that  undergoing  a  cervical  smear  and  mammogram  every 
three  years  incurs,  on  average,  $97  in  insurance  costs. 

The  last  few  studies  on  cervical  smear  cytology  reported  preventive  care  rates  and/or  claims  data.  One  study 

(Keeler,  1987)  showed  that  there  was  no  difference  in  cervical  smear  rates  between  free  care  and  cost-sharing 

care.  This  finding  was  corroborated  by  another  study  (Gherkin,  1990)  which  also  reported  no  difference  in 

cervical  smear  rates  among  patients  with  a  $5  office  visit  co-pay  compared  with  patients  with  no  co-pay.  The 

final  study  (Burton,  1997)  found  that  Part  A  Medicare  claims  for  patients  receiving  an  intervention  of  patient 

financial  incentives,  provider  financial  incentives,  and  patient  reminders  were  less  than  those  of  the  usual  care 
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group.  However,  there  were  no  differences  in  Part  B  claims,  and  a  regression  model  showed  no  differences  in 
Parts  A  or  B  claims  during  the  third  and  fourth  years  after  the  intervention. 

Colon  Cancer  Screening 

Seven  articles  on  colon  cancer  screening  also  provided  data  on  costs.  Two  of  them  were  cost-effectiveness 
studies  reporting  the  cost  per  additional  patient  screened.  The  first  cost-effectiveness  study  (Freedman,  1994) 
reported  that  returning  FOBT  cards  by  mail  without  prepaid  postage  ($1.61)  was  more  cost-effective  than 
returning  them  by  mail  with  pre-paid  postage  ($1.71),  which  in  turn  was  more  cost-effective  than  returning 
them  in  person  ($2.24).  The  second  cost-effectiveness  study  (Morrissey,  1995)  is  described  above. 

The  remaining  five  studies  on  colon  cancer  screening  reported  only  cost  data.  For  example,  one  study  (Dales, 
1979)  reported  that  a  periodic  multiphasic  health  checkup  (which  included  screening  sigmoidoscopy)  had  a 
cost  savings  that  ranged  from  $42  to  $392  per  patient  per  year  over  1 1  years.  This  benefit  was  primarily  due 
to  improved  earnings  among  patients  in  the  intervention  group.  Another  study  (Frame,  1994)  found  that  the 
cost  of  patient  reminders  was  double  that  of  provider  reminders.  A  third  study  (Belcher,  1990)  reported  that 
the  cost  of  a  health  promotion  clinic  FOBT  was  $24  per  patient.  A  fourth  study  (Thompson,  1986)  reported 
that  patient  reminders  for  FOBT  cost  $0.95  each  for  postcard  reminders,  $5.10  each  for  phone  call  reminders, 
$1.25  each  for  in-person  nurse  reminders,  and  $5.20  each  for  in-person  physician  reminders.  The  last  study 
(Winickoff,  1984)  reported  that  the  cost  of  a  computerized  audit  of  patient  preventive  care  needs  cost  $15  per 
hour  when  conducted  for  15  hours  per  month. 

CONCLUSIONS  FROM  COST-EFFECTIVENESS  STUDIES 

Taking  the  cost-effectiveness  studies  of  all  conditions  together,  computerized  provider  reminders  appear  to  be 

more  cost  effective  when  compared  with  various  types  of  patient  reminders.  This  assumes  the  existence  of  a 

computer  system,  and  does  not  include  start-up  costs.  It  is  unclear  whether  letter  or  telephone  reminders  to 

patients  are  more  cost  effective.  Full  reimbursement,  provider  reminders,  and  organizational  change  that 

allowed  nurses  to  have  a  more  active  role  in  performing  preventive  care  services  decreased  costs  and 

improved  patient  outcomes.  Academic  detailing  was  costly  and  did  not  prove  effective  in  one  study. 
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Providing  a  cooperative  health  care  clinic  (where  influenza  and  pneumococcal  vaccinations  were  part  of  the 
health  care  program)  was  cost-effective  for  caring  for  elderly  HMO  patients. 
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Question  5:  Elements  instrumental  to  the  success  of  each  intervention 
What  elements  appear  to  be  instrumental  in  determining  the  success  of  each  intervention? 

As  discussed  previously,  we  focused  our  attention  on  factors  postulated  to  be  essential  to  the  success  of  the 
interventions  as  data  limitations  prevented  us  from  conducting  a  complete  analysis  of  intensity  and  baseline 
utilization  rate  effects.  We  considered  eight  possible  factors:  use  of  social  influence,  e.g.,  opinion-leader 
involvement;  marketing  or  outreach;  high  visual  appeal  and  clarity;  collaboration  and  teamwork;  design  based 
on  needs,  barriers,  incentives,  assessments,  or  theory;  top  management  buy-in  and  support;  and  active 
learning  strategies.  Not  all  screening  settings  allowed  all  factors  to  be  modeled  due  to  data  sparseness.  We  fit 
a  similar  model  in  each  setting  to  the  intervention  component  model  discussed  in  the  context  of  the  first  two 
research  question.  The  results  are  shown  in  Tables  15  which  is  akin  to  Table  6. 

Table  15:  Effectiveness  of  Factors  to  Improve  the  Use  of  Clinical  and  Preventive  Services  presents  the  results 
of  the  meta-regression  analysis  for  immunizations,  mammography,  cervical  smear  cytology,  and  colon  cancer 
screening.  The  table  reports  the  odds  ratio  for  improvement  in  the  receipt  of  the  screening  or  preventive 
service  for  each  factor  versus  the  control  or  usual  care  situation,  adjusted  for  other  factors  and  study  level 
differences,  and  ordered  within  service  from  the  most  effective  to  the  least.  It  also  presents  the  95% 
confidence  interval.  Not  all  factors  tested  significantly  improved  screening  rates.  We  did  notice  some 
consistent  patterns  across  the  four  regressions.  For  example,  collaboration  and  teamwork  was  the  most 
positive  factor  when  compared  with  the  control  or  usual  care  setting  for  all  services  except  mammography. 
Design  based  on  theory,  etc.  was  also  significant  for  every  service  and  had  a  relatively  consistent  effect. 
Other  factors  did  not  show  consistent  patterns. 
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Educational  interventions  effectiveness  by  year 

The  success  of  an  intervention  designed  to  increase  the  utilization  of  a  preventive  service  may  be  affected  by 
the  baseline  rate  of  utilization  for  the  service.  For  example,  when  a  service  is  offered  for  the  first  time 
(baseline  rate  =  0)  it  is  relatively  easy  to  attract  users  because  the  entire  eligible  population  is  being  targeted. 
However,  if  a  service  has  been  offered  for  many  years  and  utilization  is  high,  it  may  be  a  real  challenge  to 
recruit  the  few  remaining  individuals  because  they  are  the  most  difficult  populations. 

Since  baseline  utilization  data  was  not  available  for  all  studies,  we  used  year  of  publication  as  a  proxy.  We 
hypothesized  that,  as  time  progressed,  and  as  widespread  knowledge  and  acceptance  of  the  preventive 
services  increased,  interventions  involving  education  (both  patient  and  provider)  would  be  less  successful 
because  the  majority  of  the  target  population  had  already  been  reached.  To  measure  intervention 
effectiveness,  we  used  Number  Needed  to  Treat  (NNT).  Lower  NNTs  indicate  more  effective  interventions; 
negative  NNTs  indicate  that  the  control  group  did  better  than  the  intervention  group.  Figures  six  through  nine 
display  the  NNTs  for  each  educational  intervention  (compared  with  control  group)  by  publication  year. 
Events  that  may  have  increased  knowledge  about  a  particular  service  are  marked  in  red  on  the  timeline  and 
described  in  a  box  below  the  figure. 

Figure  six  shows  the  effectiveness  of  patient  and  provider  education  on  the  utilization  of  vaccines  for 
influenza  and  pneumococcal  pneumonia.  Educational  interventions  regarding  pneumococcal  vaccine  have 
become  less  successful  over  time.  A  similar  trend  can  be  seen  for  influenza  vaccine.  Figure  seven  displays  a 
similar  trend  for  interventions  designed  to  increase  the  use  of  mammography  through  both  patient  and 
provider  education. 

Regarding  cervical  smear  cytology  and  colon  cancer  screening,  trends  were  less  clear.  Data  on  these  studies 
are  displayed  in  Figures  eight  and  nine  respectively. 
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LIMITATIONS 

The  primary  limitation  of  this  systematic  review,  which  is  common  to  all  such  reviews,  is  the  quantity  and 
quality  of  the  original  studies.  Even  more  so  than  reviews  of  single  therapies  for  single  conditions  (i.e., 
coronary  revascularization  for  coronary  artery  disease;  pharmaceutical  therapy  for  rheumatoid  arthritis),  the 
studies  presented  here  are  extremely  heterogeneous  in  terms  of  both  the  interventions  that  were  tested  and  the 
specific  populations  or  health  care  systems  being  studied.  Furthermore,  many  of  the  study  level  variables  are 
highly  idiosyncratic  and  inter-correlated  (e.g.,  a  study  of  patient  and  provider  education  with  nurse  standing 
orders  is  also  a  study  of  mammography  use  in  low-income  African-American  women).  This  correlation 
between  intervention  level  variables  and  population  makes  the  assessment  of  the  effect  of  the  individual 
components  challenging.  Due  to  sample  size  limitations,  we  were  only  able  to  model  intervention 
components  as  main  effects  in  our  meta-regression  analysis.  As  a  result,  we  can  not  distinguish  the 
independent  effects  of  various  other  intervention  components. 

We  also  made  no  attempt  to  give  greater  importance  to  studies  that  had  better  design  and  certain 
characteristics  that  have  been  postulated  to  give  more  valid  results.  This  was  because  for  these  type  of  studies 
there  is  a  lack  of  empirical  evidence  between  study  characteristics  and  bias. 

This  study  is  also  limited  by  the  inability  of  our  analysis  to  account  for  the  clustering  of  patients  in  those 
studies  that  allocated  interventions  at  the  provider  or  community  level.  Failing  to  account  for  clustering  will 
tend  to  underestimate  the  variance  (and  hence  overestimate  the  statistical  significance)  of  our  results. 
However,  rejecting  all  studies  that  did  not  account  for  clustering  would  have  meant  rejecting  more  than  one- 
third  of  the  available  study  sample.  We  decided  to  include  such  studies  as  a  result,  and  acknowledge  that 
while  our  resulting  estimates  of  the  intervention  effect  are  probably  unbiased,  we  have  probably 
underestimated  the  uncertainty  in  those  estimates. 
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Finally,  this  study  assumes  that  interventions  will  achieve  equal  success  when  targeted  towards  adults  of  any 
age.  Our  expert  panel  stated  this  assumption  on  March  17,  1999.  We  were  not  able  to  empirically  test  this 
assumption  due  to  insufficient  data.  Most  of  the  interventions  for  pneumococcal  and  influenza  vaccines  were 
targeted  toward  persons  over  65  years  of  age,  while  most  cancer  screening  interventions  targeted  adults  under 
age  65. 
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CONCLUSIONS 

Keeping  in  mind  the  limitations  noted,  and  that  knowledge  of  local  barriers  and  opportunities  to  improve 
services  is  likely  to  be  a  key  ingredient  in  designing  effective  interventions,  we  draw  the  following 
conclusions  from  the  literature: 

1 .  Organizational  change  and  financial  incentives  are  the  interventions  that  were  most  consistent  at 
producing  the  largest  improvements  in  use  of  all  preventive  and  screening  services. 

2.  Patient  reminders  are  also  consistently  effective  across  all  preventive  and  screening  services,  although  in 
general,  they  were  less  so  than  organizational  change  or  financial  incentives.  Patient  reminders  that  are 
personalized  or  signed  by  the  patient's  physician  are  more  effective  than  reminders  that  are  generic. 

3.  Provider  reminders  are  very  effective  at  improving  receipt  of  immunizations  and  show  consistent  but 
moderate  effectiveness  at  improving  the  use  of  cancer  screening  services. 

4.  Patient  education  is  consistently  less  effective  than  organizational  change  and  reminders.  The  effect  of 
patient  education,  while  still  significant,  is  modest. 

5.  Feedback  is  of  limited,  if  any,  effectiveness. 

6.  Mass  mailings  by  PROs  to  improve  influenza  immunizations  have  been  shown  to  produce  clinically 
trivial  effects  when  unaccompanied  by  other  interventions. 

7.  Multiple  interventions  are  more  effective  than  single  interventions,  although  highly  successful  single 
interventions  exist.  Adding  organizational  change  or  reminders  to  an  "average"  intervention  produces  the 
greatest  increase  in  effectiveness.  However,  the  relative  cost  effectiveness  of  adding  interventions  has  not 
been  established. 

8.  Computer-assisted  provider  reminders  are  more  cost-effective  than  patient  reminders  in  the  few  studies 
that  have  addressed  this  issue. 
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9.  There  are  insufficient  data  to  draw  conclusions  about  which  interventions  are  most  effective  for  special 
populations,  geographic  settings,  or  delivery  systems. 

10.  There  are  insufficient  data  to  draw  conclusions  about  the  effect  of  pre-intervention  rates,  intensity  of 
interventions,  or  other  factors  in  determining  the  success  of  interventions. 
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Title:  1996  District  of  Columbia  Influenza  Immunization  Project 
State  Code:  DC 
Version:  2 
Date:  12/16/97 

PRO  Name:  Delmarva  Foundation  for  Medical  Care,  Inc. 
Condition:  Mammogram 

Title:  District  of  Columbia  Mammography  1997  Project 
State  Code:  DC 
Version:  2 
Date:  10/31/98 

PRO  Name:  Delmarva  Foundation  for  Medical  Care,  Inc. 
Condition:  Mammogram 

Title:  Improving  Preventive  Eye  Care  Utilization  in  African-American  Diabetics  in  the  District  of  Columbia 
State  Code:  DC 
Version:  4 
Date:  12/10/97 
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PRO  Name:  West  Virginia  Medical  Institute 
Condition:  Influenza 

Title:  Influenza  Vaccination  for  African-American  Beneficiaries  in  New  Castle  County 
State  Code:  DE 
Version:  2 
Date:  9/23/97 

PRO  Name:  West  Virginia  Medical  Institute 
Condition:  Mammogram 

Title:  Barriers  to  Use  of  Preventive  Services  by  Delaware  Seniors 
State  Code:  DE 
Version:  3 
Date:  12/29/97 

PRO  Name:  West  Virginia  Medical  Institute 
Condition:  Mammogram 

Title:  Factors  Influencing  Variations  in  Rates  of  Breast-Conserving  Therapy  for  Carcinoma  of  the  Breast 
State  Code:  DE 
Version:  2 
Date:  10/31/97 

PRO  Name:  Florida  Medical  Quality  Assurance,  Inc 

Condition:  Influenza,  Pneumococcal 

Title:  HMO  Immunizations  for  Influenza 

State  Code:  FL 

Version:  3 

Date:  4/24/97 

PRO  Name:  Florida  Medical  Quality  Assurance,  Inc. 

Condition:  Influenza,  Pneumococcal 

Title:  Pneumococcal  Pneumonia  Immunization/ESRD 

State  Code:  FL 

Version:  4 

Date:  12/16/96 

PRO  Name:  Florida  Medical  Quality  Assurance,  Inc 

Condition:  Influenza 

Title:  HMO/Influenza,  1995 

State  Code:  FL 

Version:  10 

Date:  12/11/97 

PRO  Name:  Florida  medical  Quality  Assurance,  Inc. 

Condition:  Influenza,  Pneumococcal 

Title:  HMO/Pneumococcal  Pneumonia  Vaccine,  1995 

State  Code:  FL 

Version:  8 

Date:  12/11/97 


184 


PRO  Name:  Florida  Medical  Quality  Assurance,  Inc 
Condition:  Influenza 

Title:  Influenza  Immunization  -Senior  Nutrition  Centers 
State  Code:  FL 
Version:  2 
Date:  4/14/97 

PRO  Name:  Florida  Medical  Quality  Assurance,  Inc 
Condition:  Influenza 

Title:  1996  Inpatient  Influenza  Immunization 
State  Code:  FL 
Version:  2 
Date:  11/26/97 

PRO  Name:  Georgia  Medical  Care  Foundation 
Condition:  Influenza 

Title:  Acute  Care  Hospital  Influenza  Quality  Improvement  Project 
State  Code:  GA 
Version:  1 
Date:  12/4/96 

PRO  Name:  Georgia  Medical  Care  Foundation 
Condition:  Influenza 

Title:  1997  Statewide  Radio  Project  to  Improve  the  Rates  of  Influenza 
State  Code:  GA 
Version:  2 
Date:  10/5/98 

PRO  Name:  Georgia  Medical  Care  Foundation 
Condition:  Influenza 

Title:  1996  Statewide  Radio  Project  to  Improve  the  Rates  of  Influenza 
State  Code:  GA 
Version:  1 
Date:  1/7/98 

PRO  Name:  Georgia  Medical  Care  Foundation 

Condition:  Influenza 

Title:  Flu  Pharmacy  Education  Project 

State  Code:  GA 

Version:  4 

Date:  12/16/97 

PRO  Name:  Georgia  Medical  Care  Foundation 
Condition:  Influenza 

Title:  1996  Radio  Project  to  Improve  the  Rates  of  Influenza  Immunization  in  County,  Georgia 
State  Code:  GA 
Version:  2 
Date:  1/7/98 
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PRO  Name:  Georgia  Medical  Care  Foundation 
Condition:  Influenza 

Title:  Horizon  Flu  Project  Year  1  (Historically  Black  Colleges  and  Universities) 
State  Code:  GA 
Version:  6 
Date:  12/22/97 

PRO  Name:  Georgia  Medical  Care  Foundation 
Condition:  Influenza 

Title:  Horizon  Pilot  Project  -  Year  2  Georgia  -  African  American  Targeted  Direct  Mail  Influenza 
State  Code:  GA 
Version:  2 
Date:  10/5/98 

PRO  Name: 

Condition:  MAMMOGRAPHY 
State  Code:  ID 
Version:  1 
Date:  12/12/96 

PRO  Name:  Iowa  Foundation  for  Medical  Care 
Condition:  Mammogram 

Title:  1995-1996  Mammography  Screening  Project 
State  Code:  I A 
Version:  9 
Date:  9/30/97 

PRO  Name:  Illinois  Foundation  for  Quality  Health  Care 
Condition:  Influenza 

Title:  1996  Influenza  Immunization  Project  ~  Fee 
State  Code:  IL 
Version:  5 
Date:  9/25/98 

PRO  Name:  Health  Care  Excel 

Condition:  Mammogram 

Title:  Breast  Conserving  Surgery 

State  Code:  IN 

Version:  1 

Date:  7/10/95 

PRO  Name:  Health  Care  Excel,  Incorporated 
Condition:  Influenza 

Title:  Improving  Influenza  Immunization  Rates  in  the  Evansville  Community 
State  Code:  IN 
Version:  5 
Date:  2/24/98 
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PRO  Name:  Health  Care  Excel 

Condition:  Mammogram 

Title:  Breast  Conserving  Surgery 

State  Code:  KY 

Version:  1 

Date:  7/10/95 

PRO  Name:  Health  Care  Excel,  Incorporated 
Condition:  Influenza 

Title:  Influenza  Immunization  in  Collaboration  with  Kentucky  State  University  (Horizons  Project) 
State  Code:  KY 
Version:  6 
Date:  6/30/98 

PRO  Name:  Louisiana  Health  Care  Review,  Inc. 

Condition:  Influenza 

Title:  Influenza  Immunization  96 

State  Code:  LA 

Version:  2 

Date:  09/12/97 

PRO  Name:  MassPRO 
Condition:  Influenza 

Title:  Improving  Influenza  Immunization  in  Managed  Care 
State  Code:  MA 
Version:  3 
Date:  3/11/98 

PRO  Name:  MassPRO 
Condition:  Influenza 

Title:  Improving  influenza  immunization  in  managed  care 
State  Code:  MA 
Version:  3 
Date:  3/11/98 

PRO  Name:  MassPRO 
Condition:  Influenza 

Title:  1997  Achieving  Excellence  in  Hospital-based  Influenza 
State  Code:  MA 
Version:  4.0 
Date:  5/20/98 

PRO  Name:  Massachusetts  Peer  Review  Organization 
Condition:  Influenza 

Title:  Hospital-Based  Influenza  Vaccination  Project 
State  Code:  MA 
Version:  2.0 
Date:  05/27/98 
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PRO  Name:  MassPRO 

Condition:  Influenza,  Pneumococcal 

Title:  Pneumococcal  Vaccination  During  Hospitalization 

State  Code:  MA 

Version:  2.00 

Date:  3/30/98 

PRO  Name:  Delmarva  Foundation  for  Medical  Care,  Inc 

Condition:  Influenza 

Title:  Flu  1995 

State  Code:  MD 

Version:  2 

Date:  8/15/97 

PRO  Name:  Delmarva  Foundation  for  Medical  Care,  Inc. 
Condition:  Mammogram 

Title:  Maryland  Mammography  Screening  Focused  County  Project 
State  Code:  MD 
Version:  4 
Date:  10/19/98 

PRO  Name:  Delmarva  Foundation  for  Medical  Care,  Inc. 
Condition:  Influenza 

Title:  1996  Maryland  Influenza  Immunization  Project 
State  Code:  MD 
Version:  2 
Date:  11/28/97 

PRO  Name:  Delmarva  Foundation  for  Medical  Care,  Inc. 

Condition:  Influenza 

Title:  Maryland  Flu  1997 

State  Code:  MD 

Version:  4 

Date:  10/23/98 

PRO  Name:  Delmarva  Foundation  for  Medical  Care,  Inc. 

Condition:  Influenza 

Title:  Horizons  1997 

State  Code:  MD 

Version:  3 

Date:  10/21/98 

PRO  Name:  Northeast  Health  Care  Quality  Foundation 
Condition:  Influenza 

Title:  Increasing  the  Influenza  Immunization  Rates  in  Franklin  and  Washington  Counties 
State  Code:  ME 
Version:  2 
Date:  10/31/97 
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PRO  Name:  Northeast  Health  Care  Quality  Foundation 
Condition:  Influenza 

Title:  Increasing  the  Influenza  Immunization  Rate  in  Penobscot  County 
State  Code:  ME 
Version:  4 
Date:  09/21/98 

PRO  Name:  Northeast  Health  Care  Quality  Foundation 
Condition:  Influenza 

Title:  Increasing  the  Influenza  Immunization  Rate  in  Maine  Through  Health  Care  Providers 
State  Code:  ME 
Version:  6 
Date:  9/21/98 

PRO  Name:  Michigan  Peer  Review  Organization  (MPRO) 
Condition:  Influenza 

Title:  Increasing  Influenza  Immunization  Among  African-Americans  in  Wayne  County,  Michigan 
State  Code:  MI 
Version:  6 
Date:  9/28/98 

PRO  Name:  Michigan  Peer  Review  Organization  (MPRO) 
Condition:  Influenza 

Title:  Influenza  immunization  of  Medicare  patient  on  hospital  discharge:  a  missed  opportunity  quality 
improvement  project 
State  Code:  MI 
Version:  2 
Date:  7/29/96 

PRO  Name:  Michigan  Peer  Review  Organization  (MPRO) 
Condition:  Influenza 

Title:  Improvement  of  Rates  of  Influenza  Among  Medicare  Patients  Enrolled  in  HMOs  in  Michigan 

State  Code:  MI 
Version:  2 
Date:  12/18/97 

PRO  Name:  Michigan  Peer  Review  Organization 

Condition:  Influenza 

Title:  ESRD  Immunization 

State  Code:  MI 

Version:  1 1 

Date:  10/1/98 

PRO  Name:  Michigan  Peer  Review  Organization 
Condition:  Influenza 

Title:  Improving  Influenza  Immunization  Rates  in  Primary  Care  Practice:  A  Collaboration  with  a  Primary 
Care  Research  Network 
State  Code:  MI 
Version:  3 
Date:  12/30/97 
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PRO  Name:  Michigan  Peer  Review  Organization  Center  for  Health  Outcomes  and  Evaluation 
Condition:  Influenza 

Title:  Family  Practice  Residency  Influenza  Project 
State  Code:  MI 
Version:  4 
Date:  1 1/25/97 

PRO  Name:  Michigan  Peer  Review  Organization 
Condition:  Mammogram 

Title:  Mammography  Utilization  in  the  Upper  Peninsula 
State  Code:  MI 
Version:  2 
Date:  10/31/97 

PRO  Name:  Stratis  Health 
Condition:  Influenza 

Title:  Inpatient  Influenza  Immunization  1995-1996  (Flu_2) 
State  Code:  MN 
Version:  2 
Date:  03/31/97 

PRO  Name:  Foundation  for  Health  Care  Evaluation 

Condition:  Influenza,  Pneumococcal 

Title:  Inpatient  Influenza  Immunization  1994-1995  (Flu_l) 

State  Code:  MN 

Version:  1 

Date:  08/30/96 

PRO  Name:  Foundation  for  Health  Care  Evaluation 
Condition:  Mammogram 

Title:  Mammography  Utilization  -  Outreach  Project  #1 
State  Code:  MN 
Version:  1 
Date:  6/29/96 

PRO  Name:  Stratis  Health 
Condition:  Influenza 

Title:  Influenza  Immunization  1996-1997:  Clinic-Based 
State  Code:  MN 
Version:  2 
Date:  09/15/97 

PRO  Name:  Stratis  Health 
Condition:  Influenza 

Title:  Influenza  Immunization  1996-1997  --  Inpatient 
State  Code:  MN 
Version:  2 
Date:  08/25/97 
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PRO  Name:  Missouri  Patient  Care  Review  Foundation 

Condition:  Influenza 

Title:  Adult  Immunization  Project  -  1996 

State  Code:  MO 

Version:  8 

Date:  03/31/98 

PRO  Name:  Missouri  Patient  Care  Review  Foundation 
Condition:  Influenza 

Title:  Adult  Influenza  Immunization  Project  1997 
State  Code:  MO 
Version:  3 
Date:  8/17/98 

PRO  Name:  Missouri  Patient  Care  Review  Foundation 

Condition:  Influenza 

Title:  Influenza  Immunization  Project 

State  Code:  MO 

Version:  1 

Date:  05/01/97 

PRO  Name:  Mississippi  Foundation  for  Medical  Care 
Condition:  Influenza 

Title:  Influenza  Immunization  Project  -  Phase  I 
State  Code:  MS 
Version:  1 
Date:  9/3/96 

PRO  Name:  Mississippi  Foundation  for  Medical  Care 
Condition:  Influenza 

Title:  Influenza  Immunization  Project  -  Phase  II 
State  Code:  MS 
Version:  1 
Date:  9/3/96 

PRO  Name:  Mississippi  Foundation  for  Medical  Care 
Condition:  Influenza 

Title:  Influenza  Immunization  Project  1996-1997 
State  Code:  MS 
Version:  3 
Date:  9/23/97 

PRO  Name:  Mississippi  Foundation  for  Medical  Care 

Condition:  Influenza 

Title:  Horizons  Pilot  Project  1996 

State  Code:  MS 

Version:  3 

Date:  9/23/97 
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PRO  Name:  Montana  Wyoming  Foundation  for  Medical  Care 

Condition:  Influenza 

Title:  Influenza  Immunization 

State  Code:  MT 

Version:  1 

Date:  1 1/19/97 

PRO  Name:  Pacific  Quality  Health  Foundation 
Condition:  Pneumococcal 

Title:  Office  based  pneumococcal  vaccination  project 
State  Code:  MT 
Version:  1 
Date:  7/30/98 

PRO  Name:  Medical  Review  of  North  Carolina,  Inc 
Condition:  Influenza 

Title:  Increasing  Influenza  Immunizations  for  Medicare  Beneficiaries 
State  Code:  NC 
Version:  5 
Date:  9/2/97 

PRO  Name:  Medical  Review  of  North  Carolina,  Inc. 
Condition:  Influenza 

Title:  Increasing  Influenza  Immunizations  for  Medicare  Beneficiaries 
State  Code:  NC 
Version:  3 
Date:  2/26/98 

PRO  Name:  Medical  Review  of  North  Carolina,  Inc. 
Condition:  Mammogram 

Title:  Increased  Utilization  of  Screening  Mammography 
State  Code:  NC 
Version:  3 
Date:  3/9/98 

PRO  Name:  North  Dakota  Health  Care  Review,  Inc. 
Condition:  Influenza 

Title:  Influenza  Immunization  Mass  Immunizer  Project 
State  Code:  ND 
Version:  3 
Date:  4/14/98 

PRO  Name:  North  Dakota  Health  Care  Review,  Inc. 
Condition:  Mammogram 

Title:  Promoting  Mammography  Screening  to  Older  Women  in  North  Dakota  -  Small  Group  Discussion 
State  Code:  ND 
Version:  3 
Date:  4/14/98 
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PRO  Name:  Northeast  Health  Care  Quality  Foundation 
Condition:  Influenza 

Title:  Increasing  Influenza  Immunization  Rates  in  Coos  County 
State  Code:  NH 
Version:  4 
Date:  08/29/97 

PRO  Name:  Northeast  Health  Care  Quality  Foundation 
Condition:  Influenza 

Title:  Increasing  the  Influenza  Immunization  Rate  in  Merrimack  County 
State  Code:  NH 
Version:  3 
Date:  09/21/98 

PRO  Name:  Northeast  Health  Care  Quality  Foundation 
Condition:  Influenza 

Title:  Increasing  the  Influenza  Immunization  Rate  in  New  Hampshire  Through  Health  Care  Providers 
State  Code:  NH 
Version:  5 
Date:  09/21/98 

PRO  Name:  Northeast  Health  Care  Quality  Foundation 
Condition:  Mammogram 

Title:  Increasing  the  Utilization  Rate  for  Mammography 
State  Code:  NH 
Version:  4 
Date:  08/25/98 

PRO  Name:  Northeast  Health  Care  Quality  Foundation 
Condition:  Influenza 

Title:  Increasing  Influenza  Vaccination  Rates 
State  Code:  NH 
Version:  1 
Date:  6/1/96 

PRO  Name:  New  Mexico  Medical  Review  Association 

Condition:  Influenza 

Title:  Influenza  Immunizations 

State  Code:  NM 

Version:  6 

Date:  2/13/98 

PRO  Name:  IPRO 
Condition:  Influenza 

Title:  Inpatient  Influenza  Immunization  1997 
State  Code:  NY 
Version:  2 
Date:  7/15/98 
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PRO  Name:  Peer  Review  Systems,  Inc. 
Condition:  Influenza 

Title:  Mammography  Utilization  in  Cuyahoga  County 
State  Code:  OH 
Version:  3 
Date:  5/12/98 

PRO  Name:  Peer  Review  Systems,  Inc 
Condition:  Influenza 

Title:  1996  Seven  County  Influenza  Vaccination  Project 
State  Code:  OH 
Version:  3 
Date:  1 1/24/97 

PRO  Name:  Peer  Review  Systems,  Inc. 
Condition:  Mammogram 

Title:  Mammography  Utilization  in  5  Rural  Counties 
State  Code:  OH 
Version:  2 
Date:  12/29/97 

PRO  Name:  Peer  Review  Systems,  Inc. 
Condition:  Influenza,  Pneumococcal 

Title:  PRS/VHA  Hospital  Pneumococcal  Vaccination  project 
State  Code:  OH 
Version:  2 
Date:  8/8/97 

PRO  Name:  Oklahoma  Foundation  for  Medical  Quality 

Condition:  Mammogram 

Title:  1996-1997  Preventive  Vaccine  Project 

State  Code:  OK 

Version:  5 

Date:  3/01/98 

PRO  Name:  OMPRO 
Condition:  Influenza 

Title:  1996  Oregon  Influenza  Immunization  Campaign 
State  Code:  OR 
Version:  2 
Date:  9/30/97 

PRO  Name: 

Condition:  MAMMOGRAPHY 
State  Code:  AK 
Version:  1 
Date:  1/22/99 
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PRO  Name:  KePRO 
Condition:  Influenza 

Title:  Adult  Influenza  Immunization  Fee  for  Service  Project 
State  Code:  PA 
Version:  5 
Date:  9/30/97 

PRO  Name:  KePRO 
Condition:  Influenza 

Title:  Managed  Care  Influenza  Campaign,  1996 
State  Code:  PA 
Version:  4 
Date:  9/29/97 

PRO  Name:  KePRO 
Condition:  Influenza 

Title:  Increasing  Influenza  Immunization  Among  African  Americans 
State  Code:  PA 
Version:  3 
Date:  9/29/97 

PRO  Name: 

Condition:  MAMMOGRAPHY 
State  Code:  PR 
Version:  7 
Date:  10/21/98 

PRO  Name:  Carolina  Medical  Review 
Condition:  Influenza 

Title:  1996-1997  Inpatient  Influenza  Immunization  Project 
State  Code:  SC 
Version:  3 
Date:  11/6/97 

PRO  Name:  Carolina  Medical  Review 
Condition:  Influenza 

Title:  1996-1997  Influenza  Immunization  Initiative 
State  Code:  SC 
Version:  1 
Date:  11/15/96 

PRO  Name:  Carolina  Medical  Review 

Condition:  Mammogram 

Title:  Mammography  Screening  Campaign 

State  Code:  SC 

Version:  1 :00 

Date:  7/30/96 
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PRO  Name:  Carolina  Medical  Review 
Condition:  Influenza 

Title:  1995-96  Beneficiary  Influenza  Immunization  Outreach 
State  Code:  SC 
Version:  1 
Date:  7/18/96 

PRO  Name:  South  Dakota  Foundation  for  Medical  Care 

Condition:  Influenza 

Title:  1996  Influenza  Prevention  Project 

State  Code:  SD 

Version:  4 

Date:  8/15/97 

PRO  Name:  South  Dakota  Foundation  for  Medical 

Condition:  Influenza 

Title:  1995  Influenza  Prevention  Project 

State  Code:  SD 

Version:  1 

Date:  7/31/96 

PRO  Name:  Texas  Medical 
Condition:  Influenza 

Title:  Influenza  Immunization  in  a  Skilled  Nursing  Facility  (SNF) 
State  Code:  TX 
Version:  2 
Date:  7/1/96 

PRO  Name:  Texas  Medical  Foundation  (TMF) 
Condition:  Influenza 

Title:  Health  Care  Financing  Administration  (HCFA)  1995  Influenza  Campaign 
State  Code:  TX 
Version:  2 
Date:  7/1/96 

PRO  Name:  Texas  Medical  Foundation  (TMF) 
Condition:  Influenza 

Title:  Adelante  con  su  salud  (Hispanic  Influenza) 
State  Code:  TX 
Version:  9 
Date:  6/1/98 

PRO  Name:  Texas  Medical  Foundation  (TMF) 
Condition:  Influenza 

Title:  Improving  Influenza  Immunizations  Among  Medicare  Special  Populations 
State  Code:  TX 
Version:  3 
Date:  8/1/98 
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PRO  Name:  Virginia  Health  Quality  Center 

Condition:  Influenza 

Title:  Influenza  Immunization  1996 

State  Code:  VA 

Version:  4 

Date:  03/3 1/98 

PRO  Name:  Virgin  Islands  Medical  Institute,  Inc. 

Condition:  Mammogram 

Title:  Early  Detection  of  Breast  Cancer 

State  Code:  Virgin  Islands 

Version:  1 

Date:  1/1/95 

PRO  Name:  Virgin  Islands  Medical  Institute,  Inc 
Condition:  Influenza 

Title:  Virgin  Islands  Influenza  Campaign  1996 
State  Code:  VI 
Version:  1 
Date:  5/31/98 

PRO  Name:  Northeast  Health  Care  Quality  Foundation 
Condition:  Influenza 

Title:  Increasing  Influenza  Immunization  Rates  in  the  Northeast  Kingdom 
State  Code:  VT 
Version:  4 
Date:  08/29/97 

PRO  Name:  Northeast  Health  Care  Quality  Foundation 
Condition:  Influenza 

Title:  Increasing  the  Influenza  Immunization  Rate  in  Chittenden  County 
State  Code:  VT 
Version:  3 
Date:  09/21/98 

PRO  Name:  Northeast  Health  Care  Quality  Foundation 
Condition:  Influenza 

Title:  Increasing  the  Influenza  Immunization  Rate  in  Vermont  Through  Health  Care  Providers 
State  Code:  VT 
Version:  5 
Date:  09/21/98 

PRO  Name:  Northeast  Health  Care  Quality  Foundation 
Condition:  Mammogram 

Title:  Increasing  the  Utilization  Rate  for  Mammography 
State  Code:  VT 
Version:  4 
Date:  08/24/198 
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PRO  Name:  MetaStar 
Condition:  Influenza 

Title:  Influenza  Vaccination  (Inpatient  &  Outpatient)  and  Pneumococcal  Vaccination  (Outpatient) 
State  Code:  WI 
Version:  2 
Date:  12/1/97 

PRO  Name:  MetaStar 
Condition:  Influenza 
Title:  Influenza  '96 
State  Code:  WI 
Version:  4 
Date:  5/1/98 

PRO  Name: 

Condition:  MAMMOGRAPHY 
State  Code:  WI 
Version:  5 
Date:  3/30/99 

PRO  Name:  West  Virginia  Medical  Institute 

Condition:  Influenza 

Title:  ESRD  Influenza  Vaccination 

State  Code:  WV 

Version:  2 

Date:  10/7/98 

PRO  Name:  West  Virginia  Medical  Institute 
Condition:  Mammogram 

Title:  Factors  Influencing  Variations  in  Rates  of  Breast-Conserving  Therapy  for  Carcinoma  of  the 
State  Code:  WV 
Version:  3 
Date:  10/31/97 

PRO  Name:  West  Virginia  Medical  Institute 

Condition:  GI  Endoscopy 

Title:  Upper  and  Lower  GI  Endoscopy 

State  Code:  WV 

Version:  2 

Date:  12/30/97 

PRO  Name:  Montana  Wyoming  Foundation  for  Medical  Care 

Condition:  Influenza 

Title:  Influenza  Immunization 

State  Code:  WY 

Version:  1 

Date:  11/19/97 
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Table  16.  Expert  Reviewers 


William  Roper,  MD,  MPH 
Dean,  School  of  Public  Health 
UNC  Chapel  Hill 


Jim  Fries,  MD 
Stanford  University 


Chairman  of  Preventive  Medicine 
Kaiser  Permanente,  Denver,  CO 


Ned  Calonge,  MD,  MPH 


Steve  Jencks,  MD,  MPH 

Heath  Care  Financing  Administration 

Director  of  Quality  Improvement  Group 


Jonathan  Fielding,  MD,  MPH,  MBA 

Director  of  Public  Health 

L.A.  County  Dept.  of  Health  Services 


Michael  Siegel,  MD 
Medical  Director 
Aetna  US  Healthcare 


Lee  Newcomber,  MD 
United  Health  Group 


William  E.  Golden,  MD 
President.  AHQA 

University  of  Arkansas  for  Medical  Sciences 


Tom  Lee,  MD 

Partners  Healthcare  System  Incorporated 


Jessie  Gruman,  PhD 
Executive  Director 

Center  for  the  Advancement  of  Health 


Richard  Besdine,  MD,  FACP 
Professor  of  Medicine 
U  Conn  Center  on  Aging 
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GLOSSARY  OF  TERMS 

Intervention  Types 

Education.  Medical  education  interventions  span  a  broad  array  of  methods  for  disseminating  information  to 
health  professionals,  including  distribution  of  published  or  printed  recommendations  for  clinical  care,  such  as 
clinical  practice  guidelines,  audio-visual  materials,  and  electronic  publications.  Materials  may  be  delivered 
personally  or  through  mass  mailings.  Providers  may  also  attend  conferences,  workshops,  trainings,  or 
lectures.  Patients  may  be  educated  through  pamphlets,  peer  educators,  newsletters,  and  so  forth. 

Feedback.  Feedback  occurs  when  a  summary  of  clinical  performance  over  a  specified  period  of  time  is  given 
to  a  provider.  The  summary  may  include  recommendations  for  clinical  action.  The  information  may  have 
been  obtained  from  medical  records,  computerized  databases,  or  observations  from  patients.  Audit  and 
feedback  are  often  combined  with  opinion  leader  counseling  of  clinicians,  and  have  the  potential  both  to 
improve  clinical  knowledge  and  to  create  social  influences  that  may  improve  performance. 

Financial  Incentives.  Direct  or  indirect  financial  reward  or  benefit  can  be  tied  to  a  specific  action  on  the  part 
of  a  provider  or  a  patient.  For  example,  patient  co-payments  may  be  eliminated  or  reduced,  or  gifts  can  be 
offered  to  patients  as  incentives. 

Media  Campaigns.  Media  campaigns  use  communication  that  reaches  great  numbers  of  people  including 
television,  radio,  newspapers,  posters,  leaflets  and  booklets,  alone  or  in  conjunction  with  other  interventions. 
Campaigns  are  usually  targeted  at  the  population  level. 

Organizational  Change.  We  defined  organizational  change  as  any  change  in  the  process  of  care  at  a  clinic, 
program,  or  hospital  specifically  designed  to  improve  preventive  care  services.  This  definition  encompasses 
adding  new  personnel  with  new  functions  to  the  process  of  care,  changing  the  process  by  which  patients 
obtain  services  or  appointments,  changing  the  role  of  a  current  provider,  or  improving  the  facilities  or 
infrastructure.  Additional  examples  of  organizational  change  include  adding  case  management  (e.g., 
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coordination  of  assessment,  treatment,  and  arrangement  for  referrals)  and  revision  of  professional  roles,  such 
as  shifting  roles  among  health  care  professionals  (i.e.,  from  doctors  to  nurses,  psychologists  to  social 
workers). 

Regulatory  &  Legislative  Interventions.  These  initiatives  are  not  often  viewed  as  healthcare  provider 
behavior  change  interventions  because  they  operate  at  a  national  or  state  level,  but  such  initiatives  can  have 
significant,  if  sometimes  indirect,  influences  on  provider  behavior.  Legislative  and  regulatory  initiatives 
operate  by  changing  the  environment  and  organizations  within  which  providers  practice,  and  by  creating  new 
incentives  and  barriers  that  shape  behavior.  Examples  include  changes  in  medical  liability,  licensure 
requirements,  and  management  of  patient  complaints. 

Reminders.  These  can  be  provided  verbally,  on  paper,  or  on  a  computer  screen.  They  can  be  intended  to 
prompt  a  health  professional  to  recall  information  or  a  patient  to  utilize  a  service.  They  can  be  mailed  to 
patients,  placed  in  charts,  or  even  sent  via  e-mail  to  providers. 
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